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Abstract. We provide formulas for the denominator and superdenominator of 
a basic classical type Lie superalgebra for any set of positive roots. We establish a 
connection between certain sets of positive roots and the theory of reductive dual 
pairs of real Lie groups, and , as an application of these formulas, we recover the 
Thcta correspondence for compact dual pairs. Along the way we give an explicit 
description of the real forms of basic classical type Lie superalgebras. 
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I. Introduction 
The Weyl denominator identity 



1.1) J] (1 - e- a ) = J2 s 9 n( 

agA+ wew 



w)e w{p) - 



is one of the most intriguing combinatorial identities in the character ring of a 
complex finite-dimensional simple Lie algebra. It admits far reaching generalizations 
to the Kac-Moody setting, where it provides a proof for the Macdonald's identities 
(see [H]). Its role in representation theory is well-understood, since the inverse 
of the l.h.s. of (II .ip is the character of the Verma module M(0) with the highest 
weight 0. 

In the first part of this paper we provide a generalization of formula ( II. ip to the 
setting of basic classical type Lie superalgebras. Deepening this problem, we came 
across an interesting connection with representation theory of Lie groups which is 
the theme of the second part of the paper. 

By a basic classical type Lie superalgebra we mean an almost simple finite- 
dimensional Lie superalgebra g = £io © Si with a non-degenerate invariant super- 
symmetric bilinear form (•, •) and Qo reductive. Choosing a Cartan subalgebra f) 
of 00) we get the set of roots A = A U where Aj is the set of roots of f) in 
Qi, i = 0, 1. Choosing a set of positive roots A + in A, we let A+ = A + D Aj. In 
trying to extend ( II. ip to a Lie superalgebra q, it is natural to replace the l.h.s of 
( II. ip with the character of the Verma module M(0) over q, which is the inverse of 

,i.2) H _iWi-o 



rL 6 A+(i+e-") 



*eA+ 

called the denominator. Beyond the denominator R, very important for us will be 
the super 'denominator, defined as 



;i.3) R 



Generalizations of formulas for R and R to affine superalgebras and their con- 
nection with number theory and the theory of special functions are thoroughly 
discussed in [H]. The striking differences which make the super case very different 
from the purely even one are the following. First, it is no more true that subsets of 
positive roots are conjugate under the Weyl group: to get transitivity on the sets of 
positive roots one has to consider Serganova's odd reflections, see (12.51) . Moreover 
the restriction of the nondegenerate invariant bilinear form (• , •) to the real span 
Va of roots may be indefinite, hence isotropic sets of roots appear naturally. One 
defines the defect d = def q as the dimension of a maximal isotropic subspace of 
Va- A subset of A, consisting of linearly independent pairwise orthogonal isotropic 
roots is called isotropic. It is known that any maximal isotropic subset of A consists 
of d roots (PH). 

In this paper we settle completely the problem of finding an analogue of (II. ip 
for basic classical type Lie superalgebras, by providing an expression for the r.h.s. 
which incorporates the dependence on the set of positive roots. The following result 
is known. 
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Theorem KWG. Let g be a basic classical type Lie superalgebra and let A + be 
a set of positive roots such that there exists a maximal isotropic subset S of A + , 
contained in the set of simple roots corresponding to A + . Then 



1.4) ef>R= Yl sgn(w)wl= ^ 

1.5) e"R = sgn'(w)w (= ^ 




In the above statement p = Po — Pi, where pi = \ J2 a &A + a ' W * s a subgroup of 
the Weyl group W g of q defined in fl2.ll) and sgn 1 is defined in (12. 3p . This result 
has been stated by Kac and Wakimoto in [T4], and fully proved for d — 1 by using 
representation theoretical methods. A combinatorial proof for arbitrary d has been 
recently obtained by Gorelik [I]. 

Theorem KWG will be of fundamental importance for our generalization, which 
is as follows: given any subset of positive roots A + , we want to express e p R by a 
formula like the r.h.s. of (11. 4p . in which at the numerator appears the p correspond- 
ing to A + and at the denominator a suitable maximal isotropic subset S of A + , so 
that the exponents at the denominator are linear combinations with non-positive 
integer coefficients of simple roots: see (ll.lOp . (II .lip . We also provide a formula in 
which the denominator is exactly as in the r.h.s. of of (ll.4p . but we have to perform 
a correction on p, depending on S: see (I1.15p . 

Let us discuss more in detail these formulas. First we construct a certain class S 
of maximal isotropic subsets by the following procedure. 

Definition 1.1. Denote by S(A + ) the collection of maximal isotropic subsets of A + 
of the form S — S% U . . . USt of A + , where Si is a non empty isotropic subset of the 
set of simple roots, and, inductively, Si is such a subset in the set of indecomposable 
roots of S^_ 1 \ Si-i, where S^ x = {a G A + | (a, f3) = OV/3 G Si_i}. 

Denote by Q, Qo the lattices spanned over Z by all roots and even roots, respec- 
tively, and let, for S G S(A + ) and 7 G S 



(1.6) e(r,) 



1 if 7] G Qo, 
1 if 77 G Q \ Qo, 

(1.7) 7- = {P e S I p < 7}, 7 < = {P e S I p < 7}, 

(1.8) [7] = E e fr " W> M= E £ ^ ~ 

(1.9) sgn^) = (-l)l^l +1 , 

where, as usual, /3 < 7 if 7 — /3isa sum of positive roots or zero and \X\ stands for 
the cardinality of the set X . 



Theorem 1.1. Let = 0o © Si be a basic classical type Lie superalgebra having 
defect d, where q = A(d — l,d — 1) is replaced by gl(d,d). Let A + be any set of 
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positive roots. For any S G 5(A + ) we have 

(1.10) C'e?R=Y, sgn(w)w ( = _ 

(1.11) C-e p R=J2 sgn'(w)w(= ^ 

where 

(1.12) C- ° s 



j-r ht(rf)+l ' 
76S 2 



and C g = |W B /W|. Moreover, inere exzsfo S nice G 5(A+) snc/i £/iai [7] G Q" 
Z+A+ /or any 7 G S™ ce . 

The explicit values of C s are the following. 

f d\ if = A{m — 1, n — 1), 

2 d d! iffl = B(m,n), 

2 d d! if = -D(m, n), m > n, 

2 d ~ 1 d\ if = D(m,n), n > in, 

1 if0 = C(n), 

2 if fl = D(2,l, a), F(4), G(3). 



;i.i3) c 



If def(fl) = 1, and 5 G *S(A + ), then 5 consists of a single simple root. Hence 
we are in the hypothesis of Theorem KWG, in which case (ll.lOp . (II. lip hold with 
C = C g , and these formulas coincide with ( II. 4p . ( II. 5p . Therefore we have to deal 
only with Lie superalgebras of type gl(m,n), B(m,n), D(m,n), which have defect 
d = min{m, n}. We also treat the case A(d — 1, d — 1), see Subsection 13.2.21 

In these cases we introduce a combinatorial encoding of the elements of 5(A + ) 
using the notion of an arc diagram (see Definition 13. ip . To any arc diagram X we 
associate a maximal isotropic set S(X), and in Proposition 13.21 we show that this 
is a bijection between all arc diagrams and S(A + ). 

We introduce two types of operations on arc diagrams: odd reflections r v and 
interval reflections r\ v w \. They have the following features: 

(1) for any arc diagram X there exists a finite sequence of odd and interval 
reflections which change X into an arc diagram X' such that S(X') consists 
of simple roots (cf. Lemma [3.4p : 

(2) we are able to relate J2 w &w s 9 n ( w ) w (-q e (i-e-M) ) ^° ^ ne srmnar sums 

where the product in the denominator of the l.h.s ranges over r v X and over 
r [vM X (cf. dSZZD)- 

The above properties allow to prove formula (11.111) starting from Theorem KWG 
applied to S(X'). In Lemma [3.71 we show that formula (11.101) can be derived from 
(II. lip . We also single out a special set of arc diagrams (see Definition 13.21) which 
have the property described in the last sentence of Theorem 11.11 

Further, we provide a different generalization of (jl.5p . in which only positive 
roots appear in the denominator, namely we have 
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Proposition 1.2. Let S G <S(A + ). Then 

_l)E 7eS l7 < l e P+E 7 g S l7[ ' 



1.14) efR= 



sgn{w)w 



[1.15) e p R= s 9 n ' 



pP+E 7 esM 



u; )w 



™ eH /# \ 11/365 

The main application of our formulas is a conceptually uniform derivation of the 
Theta correspondence for compact dual pairs. Basic classical type Lie superalgebras 
are linked to Howe theory of dual pairs through the notion of a distinguished set 
of positive roots, which we introduce in Definition 14.11 it requires that the depth 
of the grading, which assigns value 1 (resp. 0) to each generator corresponding to 
a positive odd (resp. even) simple root, does not exceed 2. The distinguished sets 
of positive roots for the basic classical type Lie superalgebras (which are not Lie 
algebras) are classified in Subsection 14.41 

To understand the relationship with Howe theory, consider the complex symplec- 
tic space ($ji, (• , ■)), where (• , ■) = (■ , -)| fll . A choice of a set of positive roots A + 
determines a polarization g 1 = gf®Qi, where gf = © Q a - Hence we can consider 

the Weyl algebra W(gi) of (jjx, (■ , •)) and construct the W(gi)-module 

(1.16) M A+ ( Ql ) = W( gi )/W( gi )gf, 

with action by left multiplication. The module M A+ ($ji) is also a sp(gi, (• , •))- 
module with T G sp(fli, (• , •)) acting by left multiplication by 

dim gi 

(1.17) 9{T) = "2 E T (^K> 

i=i 

where {x^} is any basis of gi and {x 1 } is its dual basis w.r.t. (■,•), i.e. (xj, x J ) = 6ij. 
It is easy to check that, in W(fli), relation 

(1.18) [6(T),x]=T(x) 

holds for any x G 0i. This implies that we have an f)-module isomorphism 

(1.19) M A+ ( 01 ) = S( r)®C_ pl 

where S^gj) is the symmetric algebra of and C_ P1 is the 1-dimensional f)-module 
with highest weight —p\. Hence the ()-character of M A+ (gi) is given by 

Since ad\ gi (g ) C sp(gi, (• , ■)), we obtain an action of g on M A+ (0!). Upon mul- 
tiplication by e p0 riaeA+(l — e_Q ) ^ ne r .h- s - °f (H-20p becomes e p R and equating it 
to our formula we obtain the go-character of M A+ (gi). 

So far we have not used the special features of distinguished sets of positive roots 
A + . Restricting to distinguished sets of positive roots A + for Lie superalgebras 
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of type gl(m,n), B(m,n), D(m,n), we are able to build up a real form V of Qi 
endowed with a standard symplectic basis {e a , f a } a eA+ sucn that 

C(e Q ± y/=lf a ) = at 

It turns out that sp(V) D ad\ gi (g ) — Si x s 2) s i ; i = 1,2, being the Lie algebras of 
a compact dual pair in Sp(V). As stated in Proposition 14.81 distinguished sets of 
positive roots turn out to correspond in this way to all compact dual pairs. Howe 
theory and our denominator formula allow to recover the explicit computation of 
the Theta correspondence done by Kashiwara-Vergne [15] and Li-Paul- Tan-Zhu [T7j 
and related results by Enright [2j. Before discovering formula fll.lOp . we used this 
argument the other way around, to deduce the denominator identities from the 
knowledge of the Theta correspondence: the relevant details are given in [T3] . 

In Section 14.81 we show that relaxing the condition on the depth of the grading 
which defines the distinguished sets of positive roots it is possible to generalize the 
previous approach to all but one noncompact type I dual pairs (cf. Proposition 
14.2]) . We plan to investigate in a subsequent paper if our methods can be pushed 
further to obtain information on the Theta correspondence in these cases too. 

In conclusion of the paper we use our superdenominator formula to confirm the 
validity of the Kac-Wakimoto conjecture [2] for the natural representations of the 
classical Lie super algebras. We intend to extend this method to a large class of 
representations in a subsequent publication. 

The paper is organized as follows. In Section [2] we collect some basic notation and 
definitions. In Section [3] we develop the combinatorial machinery needed to prove 
our superdenominator formulas: we introduce arc diagrams and study the effect of 
odd and interval reflection on arc diagrams. We finally prove the superdenominator 
formula for the non-exceptional basic classical type Lie superalgebras and also prove 
the denominator formula. The final subsection is devoted to the proof of formula 
(j!.15p . In Section H] we relate the distinguished sets of positive roots to Howe theory: 
after a preliminary discussion on Cartan involutions we introduce and analyze the 
definition of distinguished set of positive roots. Then we relate distinguished sets of 
positive roots to real forms: as a byproduct of our analysis we obtain a conceptual 
proof of the classification of real forms of basic classical type Lie superalgebras which 
is in the same spirit as the classification of simple Lie algebra involutions, given in 
[TT] . This classification has been previously obtained by Kac, Parker and Serganova, 
(cf. [9], [18], [21] ). Finally we provide a concise discussion about certain sets of 
positive roots related to noncompact dual pairs. In the follwoing four Sections we 
deduce from our denominator formula the explicit form of the Theta correspondence 
for all compact dual pairs. It is worthwhile to note that the pair (0(2m), Sp(2n, R)) 
has additional complications which require to use a version of the character formula 
for disconnected compact groups due to Kostant [16] . In Section [9] we prove the 
Kac-Wakimoto conjecture for the natural representation. 

The ground field is C throughout the paper, unless otherwise specified. 

2. Setup 

In this section we collect some notation and definitions which will be constantly 
used throughout the paper. Let q be a basic classical type Lie superalgebra. This 
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means that g = Qq © $ji is an almost simple finite-dimensional Lie superalgebra with 
reductive Qq and that g admits a nondegenerate invariant supersymmetric bilinear 
form (•,•). Almost simple means that [g,g] modulo its center is simple. Super- 
symmetric means that (gofji) = and the restriction of (■ , •) to go (resp. gi) is 
symmetric (resp. skewsymmetric). A complete list of the simple ones consists of 
four series A(m,n), B(m,n), C(n), D(m,n) and three exceptional Lie superalge- 
bras D(2, l,a), F(4), G(3), and the non-simple ones are obtained by adding to a 
simple one or to gl(n + 1, n + 1) a central ideal. These Lie superalgebras can be con- 
structed, starting with a Cartan matrix, like simple finite-dimensional Lie algebras, 
though inequivalent Cartan matrices may correspond to the same algebra (see [H]). 

Choose a Cartan subalgebra f) C go, and let A, A , Aj C f)* be the set of roots, 
even roots, odd roots, respectively. Let W s C GL(f)*) be the group generated by 
the reflections s a w.r.t. even roots a G Ao- Choose a set of positive roots A + C A 
and set A+ = A* n A + , i = 0,1. Let II be the set of simple roots corresponding to 
the choice of A + , i.e. the set of indecomposable roots in A + . Set also, as usual, for 
i = 0, 1, p.i = \ J2 a &A + a ' P = Po ~ Pi- From time to time we will write p(A + ) or 
p(n) to emphasize the dependence of p from the choice of the set of positive roots. 

Next, recall the notation skipped in the Introduction. Let h y be the dual Coxeter 
number of g, i.e. 2h y is the eigenvalue of the Casimir operator of (g, (• , •)) in the 
adjoint representation. If h y ^ (which holds unless g is of type A(n, n), D(n+1, n) 
or ,0(2,1, a)), we let [H] 

A$ = {a e A | /i v (a,a) > 0}, 

(2.1) vr« = (^eW fl |/3e A}), 

We refer to [HI Remark 1.1, b)] for the definition of W* when h y = 0. Set 

(2.2) A = {a e A | \a <£ A}, A a = {a e Ai | (a, a) = 0}. 
Finally, for w G W s , set 

(2.3) sgn(w) = (-1)^, sgn'(w) = (-l) m , 

where I is the usual length function on W s and m is the number of reflections from 
A^ occurring in an expression of w. Note that 

w (e p R) = sgn'(w) e p R. 

In particular, sgn' is well-defined. Note also that sgn = sgn' in types A and D. 

Let now recall the definition of odd reflections [19]: for an isotropic root a G II 
we define 

(2.4) r Q (A + ) = (A + \{a})U{-a}. 

It is easy to prove that r a (A + ) is a set of positive roots for g and that if we set 

ia + P if (a, ^0 

(2.5) r a (P) = h if (a,/3) = 0, a ^ (3 

I —a if /3 = a 

for /3 G IT, then r a (If) = {r a (f3) \ (3 G 11} is the corresponding set of simple roots. 
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The importance of odd reflections lies in the fact that, up to W^-action, any 
two sets of positive roots can be obtained one from the other by applying a finite 
sequence of odd refections. Note that, for an isotropic a, 

(2.6) p(r a (A + )) = p(A + ) + a, 
from which one deduces that 

(2.7) e" (A+) R(A + ) = - e ^( A+ )) R(r a (A + )) Va G II. 

3. Denominator formulas 



3.1. Arc diagrams. Let g be a Lie superalgebra of type gl(m,n), B(m,n),C(m), 
D(m,n). We first need an encoding of the sets of positive roots. The explicit real- 
izations of these Lie superalgebras given in [9] , [lOj Section 4] leads to a description 
of their roots in terms of functionals 6j, S{ G fi*. 
We let 

£ = {€!,..., e m }, V — {Si, . . . , S n } 
if g is of type gl(m, n), B(m, n) and let 

£ — { e i ; • • • ) fm-i, £m} or S = {ei, . . . , e m _i, — e m }, T> = {Si, . . . , S n } 
if q is of type C(m) or D(m,n). Let 

B = £UV. 

Then B is a basis of h* (for C(m) one has n — 1). We call two elements Vi,v^ G i3 
elements of the same type if {^1,^2} C S or {t>i,t>2} C V and elements of different 
types otherwise. 

Recall the structure of A in terms of B: in type gl(m,n) we have 

A = {e,i -e j \l<i^j<m}U{S i -S j \l<i^j< n}, 

Ai = {±(q - Sj) I 1 < i < m, 1 < j < n}. 

As an invariant bilinear form on gl(m,n) we choose the supertrace form (a, b) = 
str(ab), str being the supertrace of a matrix in gl{m,n), so that 



Si 



■>ij — (Si, Sj). 

In the other classical types see (15.11) . (16. ip for the description of roots; the invariant 
bilinear form is the restriction of the supertrace form. 

Given a total order > on B = {£1 > . . . > £ m + n }, we define a set of simple roots 
II(£>, >) for g as follows 






U(B,>) 


gl(m, n) 


{6 - 6+ 


B(m, n) 


{6-6+ 


C(m) 


{6-6+ 




{6-6+ 


D(m, n) 


{6-6+ 




{6 - 6+ 



1 m+n— 1 

fi=l 

1 m+n— 1 

U=l 



U {6ra+n} 



1 m+n— 1 
U=l 

1 m+n— 1 



U {2£ m + n } if £ m + n G 



U {Cm+n-1 + £m+n} if £m+n G 

Using Kac's description of Borel subalgebras (see j9]), it is not difficult to prove the 
following result. 
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• X • X X • • X • 

e 1 5i e 2 5 2 5 3 e 3 e 4 5 4 e 5 
Figure 1. An arc diagram X for gl(5,A). 

Lemma 3.1. Up to W s -equivalence, any set of positive roots for q has H(B, >) as 
a set of simple roots for some total order > on B. 

It may happen in type D(m,n) that two different choices of B give rise to the 
same set of simple roots II, so we make the following choice. Let h G t) be such that 
a{h) = 1 for each a G II. Choose B = {ei, . . . , e m } U {Si, . . . , S n } if e m (h) > 0, and 
B = {ei, . . . , -e m } U {5i, . . . , 5 n } if e m f» < 0. 

Instead of the Dynkin diagram, we encode II(£>, >) by the ordered sequence 
B, which is pictorially represented as an array of dots and crosses, the former 
corresponding to vertices in £ and the latter to vertices in T>. 

Examples for gl(5, 4), D(A, 3) are given in Figures 1, 2, 3 disregarding the arcs. 
In the first case, we start from the total order {ex > Si > e 2 > 5 2 > S 3 > e 3 > e 4 > 
64 > e 5 }, which corresponds to the set of simple roots {ei — Si, Si — e 2 , e 2 — S 2 , S 2 — 

83,83 ~ e 3,e3 - £4, £4 - 

In Figure 2, we start from the total order ei > Si > e 2 > S 2 > e 3 > 5 3 > e 4 , which 
gives rise to the set of simple roots {ei—Si, Si—e 2 , e 2 —S 2 , S 2 —e 3 , e 3 — S3, 5 3 ±e 4 }, while 
in Figure 3 the total order is ei > Si > e 2 > S 2 > e 3 > — e 4 > 5 4 , which corresponds 
to the set of simple roots {e 4 — Si, Si — e 2 , e 2 — S 2 , S 2 — e 3 , e 3 + e 4 , — e 4 ± 5 3 }. 

For v, w G if v > w, let [u, w] = {u e B\ v > u > w}. If B' C B, we denote 
by Wjg' the subgroup of W Q consisting of (non-signed) permutations of £>' D £ and 
of B 1 n £>, so that we have W B > = S k x S f , where |F n f | = k, \B' n D| = /. 

We now introduce arc diagrams. These are combinatorial data that encode some 
maximal isotropic subsets of the set of positive roots. 

Definition 3.1. An arc diagram is the datum consisting of the ordered sequence 
of vertices representing B, and of arcs between some of the vertices, satisfying the 
following properties: 

(i) the vertices at the ends of each arc are of different type; 

(ii) the arcs do not intersect (including the end points); 

(Hi) for each arc vw the interval [v,w] contains the same number of elements of 
£ and ofV: \[v,w]n£\ = \[v,w}nV\; 
(iv) the number of arcs is min(m,n). 

We also define the support of an arc diagram X as 

Supp(X) = {v G B I v is an end of an arc in X}. 

If A + is the set of positive roots corresponding to Yi{B, >), we denote by A(A + ) 
the set of arc diagrams whose underlying set of vertices is B and the underlying 
total order is >. 
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• X • X • X • 

e 1 5i e 2 S 2 e 3 S 3 e 4 
Figure 2. An arc diagram for D(A, 3). 



• X • X • • X 

ex 5i e 2 5 2 e 3 -e 4 S 3 
Figure 3. Another arc diagram for -D(4, 3). 

3.1.1. The maximal isotropic set S(X). Consider an arc diagram X. It encodes the 
following isotropic set of positive roots: 

S(X) = {v — w\ v,w G B are connected by an arc and v > w}. 

For example, the arc diagram in Figure 1 encodes the isotropic subset S = {Si — 
£2, £1 — S 2 , S3 — e 3 , o~ 4 — e 5 }. The arc diagram in Figure 2 encodes the isotropic subset 
S = {€1 — 61,62 — 63, £3 — 64}, whereas that in Figure 3 encodes the isotropic subset 
S' = {ei - Si, S 2 - e 3 , -63 - e 4 }. 

We will often denote by A+ (X), A+(X), A+(X), U(X) the sets of positive, posi- 
tive even, positive odd, and simple roots associated to the ordering of vertices of B 
underlying X (the arc structure of X is irrelevant for that). One readily sees that 
S(X) C A + (X) and S(X) is a basis of a maximal isotropic subspace in h* (since 
the cardinality of S(X) equals the defect of g). This shows that S(X) is indeed a 
maximal set of isotropic roots. 

For each G Ai let 

m 

sn/3 = (/3,^e,) 

i=l 

(that is sn(ej ± 6j) = 1, sn(— e, ± 6j) = —1). For j3 G S(X) one has sn/3 = 1 (resp., 
— 1) if the left end of the corresponding arc is in £ (resp., in T>). For each a G S(X) 
we have (cf. ( QD ) 

(3.1) [a] = sna-sn/3-/3, 

/3eS(X),j3<a 

namely 

E ^- E ^. i^-^-h E ^- E ^ 

E.g., with reference to Figure 1, we have 

[ei - 6 2 j = ei + e 2 - Si - 6 2 , {Si - e 2 J = Si- e 2 . 
Notice that for an arc vw the element [u — w] is VF[„ jtu ] -invariant. 

3.1.2. Special arc diagrams. 

Definition 3.2. IFe say i/iai an arc is simple if it connects consecutive vertices. 
We call an arc diagram simple if each arc is simple. 

We call an arc diagram X nice if for each a, /3 G S(X) such that a < (3 one has 
sn a = sn /3 ( equivalently, if for any arc, its left end is of the same type as the left 
ends of all arcs which are below this arc). 
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One readily sees that simple diagrams correspond to the case when S(X) con- 
sists of simple roots, and nice arc diagrams correspond to the case when [7] = 
YlpeS(x) /3<7 I* 1 particular, for a nice arc diagram X, [7] £ Q + for each 7 £ S(X). 
We denote by .4. mce (A + ) the set of nice arc diagrams X such that A + = A + (X). 

3.1.3. Existence of arc diagrams and of nice arc diagrams. Fix an order > on B, set 
n = H(B, >) and let A + be the corresponding set of positive roots. Arc diagrams 
can be easily constructed inductively. We start by drawing an arc between any two 
consecutive vertices v, w of different types. Then we throw away these two vertices 
and obtain an ordered set 8' U V, where the cardinality of 8 ' (resp., of V) is less 
by one than the cardinality of 8 (resp. of T>). We take any arc diagram for 8' U T>' , 
and we let S(X') be the set of its arcs. Then £(A) = S(X') U {vw} is the set of 
arcs of an arc diagram X for B. 

More generally, we can start by drawing any possible arc (i.e., the arc between 
two vertices v, w of different types such that [v, w] has the same numbers of vertices 
lying in 8 and in T>). Then we can construct an arc diagram for the sequence which 
is below the arc (i.e., for [v,w] \ {v, w}) and for the ordered set B \ [v,w]. The set 
of arcs in the resulting arc diagram is the union of the sets of arcs in these two 
diagrams and the arc vw. As a result, there is an arc diagram X such that /3 £ A^ 
belongs to S(X) iff the interval defined by (3 (i.e., [v, w] for = v — w) contains the 
same numbers of vertices lying in 8 and in T>. 

Look now at the above procedure from an algebraic point of view: removing a 
pair {u,v} of vertices of different type (or more generally a set Z = {uj,t>j}f =1 
of pairs of vertices of different type) from the ordered set B gives the ordered set 
corresponding to S ± \S where S = {u — v} (resp, S = {ui — v i}f =1 ). This is clearly 
related to the procedure explained in Definition 11.11 More precisely, we have 

Proposition 3.2. If X is an arc diagram, then S(X) £ 5(A + ). Viceversa, any 
S £ 5(A + ) is of the form S(X) for some arc diagram X . 

Proof. Both assertions follow easily from the remark in the previous paragraph after 
it is proved that an arc diagram has always a simple arc. To prove this, remark 
that axioms in Definition 13.11 imply that all vertices below a given arc are ends of 
some arcs. Thus the "lowest" arcs (the ones which do not have arcs below them) 
are necessarily simple. □ 

Let us now explain how to construct nice diagrams. 

Proposition 3.3. A nice (A + ) ^ 0. 

Proof. Let e\ > €2 > ■ ■ ■ > e m and Si > 62 > ■ ■ ■ > S n . Assume that e\ > S±, 
so that our sequence A = B starts with e\, . . . , €f.,5i (k > 1). We draw the arc 

oi =£k°~i and consider the sequence A\ = B \ {ek,o~i}. We repeat the procedure 
until the first element of the sequence A s+ i is not in 8. Note that the right end 
of the arc aj obtained in the jth step is 8j. Let U be the union of the ends in 
all arcs aj, j — 1, . . . , s. Let us show that U = [e\, S s ]. Indeed, by the above, for 
j = 1, . . . , s the right end of aj is 5j so both ends lie in [ei, Sj] C [e l5 S s ]. Therefore 
U C [ei,5 s ]. Moreover, Si, . . . ,S S £ U so ([ei,5 s ] \ U) C 8. Since the first element 
in A s+ i = A \U lies in V, we conclude that [ei,5 s ] = U as required. Thus the 
sequence [ei, S s ] with the ends of the arcs in U is a nice diagram X and each element 
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• • X X X • • X • 

e 1 e 2 5i 5 2 5 3 e 3 e 4 e 5 5 4 
Figure 4. r Sl - e2 (X). 



x 
Si 



x 
5 2 



X 

S3 



^2 02 <>3 ^3 e 4 

Figure 5. r[ £1)(52 ](X). 



x 

C5 



of this sequence is an end of an arc. Now we construct a nice diagram Y for the 
sequence A s+ i (which is shorter than A ). The union of X and Y is a nice arc 
diagram for the sequence Aq. □ 

3.1.4. Operations with the arc diagrams. Consider all arc diagrams on the set B 
endowed by a total order and introduce the following operations. 

Odd reflections r v _ w . Let v,w be two consecutive vertices of an arc diagram 
X connected by an arc vw. We define r v - w {X) to be the arc diagram with v,w 
switched (i.e., r v _ w (X) = (X\{vw}) U {wv}; the total order on B for r v - w (X) is 
obtained from the total order for X by interchanging v and w). Figure 4 displays 
rs 1 - e2 (X) where X is the arc diagram of Figure 1. The odd reflection on an arc 
diagram corresponds to an odd reflection with respect to a simple root lying in 
S(X) (cf. Q23D): 



n(r^(X)) =r w _ 1B (n(X)), 
5(r^(X)) = 5(X) \ {v - w} U {u; - v}. 

Notice that sna ■ a = sn(— a) ■ (—a) and thus for 7 G S(X) PI S , (r u _ tu (X)) the 
element [7] defined for X and for r v _ w (X) is the same. 

Interval reflections r[ eii d h ]- Suppose X has a subsequence ei, di, e 2 , cZ 2 , • • • , e^, dfe 
(fc > 1), where ei, . . . , are of the same type and di, . . . , df. are of another type, 

with the arcs e\d^ die 2 , d 2 e 3 , . . . , dk-iek ■ We define r\ ei)C i k ](X) as the arc diagram 
with the same total order on B, where the above arcs are substituted by the arcs 

eidi, e 2 d 2 , e 3 d 3 , . . . , e k d k . Hence 
S{r [ei , dk] (X)) = 

{S(X) \ {ei - d k , di-e 2 ,...,e k - d k }) U {e x - d x , e 2 - d 2 , . . . , e k - d k }. 

Figure 5 displays r[ eu $ 2 ](X) where X is the arc diagram of Figure 1. Figure 6 
provides further examples: the arc diagram X in the left display is not nice, since 
[ex— S 2 J = (ex— b~ 2 ) — {S\— e 2 ). The middle display represents rs 1 - e2 (X), which is nice, 
and the right display represents r[ eii s 2 ](X), which is simple (hence, in particular, 
nice). This example also shows that both odd and interval reflections are necessary 
moves to change an arc diagram into a simple one. They are also sufficient, as we 
show next. 



DENOMINATOR IDENTITIES FOR LIE SUPERALGEBRAS 



13 




• •xx • x • x 

ei e 2 Si 5 2 ei 5 1 e 2 S 2 
Figure 6. 

Lemma 3.4. There exists a finite sequence of odd and interval reflections which 
change any arc diagram into a simple arc diagram. 

Proof. We proceed by induction on the number k of non-simple arcs in the arc 
diagram X . If k = 0, then X is a simple arc diagram and there is nothing to prove. 

Assume k > 1 and let ed be a non-simple arc such that each arc ™, e > v > w > d 

is simple. Then, necessarily, we have [e, d] = ea\d x a 2 o! 2 . . . a h a' h d, with arcs ed, a^a^ 
, i — 1, . . . , h. Use the odd reflections r a ._ a /, if necessary, to modify [e, d] in such a 
way that vertices of different type alternate from e to d: call the resulting diagram 
X' . Then r[ e ^(X') has k — 1 non-simple arcs and we are done by induction. □ 

3.2. Proof of Theorem 11.11 In this Section we prove formulas fll.lOp . ( 11. lip . For 
future applications we prove a slightly more general result (see Proposition 13. 8p . 

For any subset U of W Q and any rational function Y with {e b \ b G £>} as set of 
variables, introduce the sum 

(3.2) F u (Y) = Y J S9n(w)w(Y). 
We will need also the sum 

(3.3) F u (Y) = Y J *gn\w)w(Y). 

well 

Notation 3.1. If U is a subgroup of W g , and W C W s is stable under the right 
action ofU, then W is a union of right cosets in W s /U. We denote by W/U a set 
of representatives. 

Note that 

JF W {Y) = T w/u (Tu(Y)), T W (Y) = T w/u (Tu{Y)). 

The formula displayed on the right follows since sgn' : W s — > {±1} is a homomor- 
phism. 

Lemma 3.5. Let U be a set of simple roots for g = gl(k\k), k > 2, all of which are 
isotropic. Let {/?x, • • • , fik} C II be the maximal isotropic subset. Then 

1 1 — e~ ^ - 1 

(3.4) JF Wa ( . — ) = JV B ( k — ) ■ 

n^aC 1 — e ft ) k n i= i(i-e ) 

Proof. Let A = n *_ (i- e -^) ' x ° = ^w t iA) and for j = 1, . . . , k let 



x 



j 



Then the left-hand side of formula (13.41) is equal to x — X\. LHS = x — X\. 

Write II = {ei — — e 2 , . . . — 6~k} and = €i — 5i. Observe that W g 
contains a subgroup permuting the /3j's, and all elements of this subgroup are even 
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(the permutations (e» €j)(Si 5j) switches (3i and f3j). Since JF Wsj (A') is W g -skew- 
invariant for any rational function A', we conclude that 

(3.5) Xj = P Wi (Ae~^ Pi ) 

for any subset J C {1,2, ... ,k} of cardinality j. Since the permutation (ei e 2 ) 
stabilizes A, one has 

s 
i=l 

for s = 2,...,k. Hence, using (I3.5p . we obtain that Yfj=o(~^y \j) x j = f° r 
s — 2, . . . , k. We deduce by induction on j > 2 that Xj = j{x\ — x ) + x and thus 

LHS = xo - X! = = - e-S^^y) = 1 Z e ~^ U - f We (A) 

which is (E3D- □ 

For an arc diagram X we denote by px the element pn(x) and by Rx,Rx the 
fractions i?, R constructed for II (X), that is 

p _ n aeA ,| ( x)(i-e-) _ n^w 1 - e_a ) 



Let 

ht 7 + 1 e px 



(3.6) 7>(X) = 



2 rU<?m(l-e-M)' 

Corollary 3.6. Let X be an arc diagram and r[„ )W ] fre an interval reflection. For 
any subset W of the Weyl group which is stable under the right action of Ws upp (x) 
one has 

(3.7) Fw(V{X)) = F w {V(r [vM (X))). 

Proof. Denote by Y the arc subdiagram corresponding to the interval [v, w] and let 
Y' = r[ v>w ](Y). Then X' = r[ VjW ](X) is obtained from X by substituting Y by Y' . 
View Y, Y' as arc diagrams of glik, fc)-type. Then G = W[ VjW ] is the Weyl group of 
Y and of Y' . 

Notice that (px — Py , ct) =0 for each a G II (Y) and thus px — Py is G-invariant. 
Since [v,w] C Supp(X), we have that IVG = W. Therefore 

pPX-pY 

f w {v{x)) = jf w/g {- -j- G (p(y))) 

ll/3GS'(X)\5(y)l i e J 

and a similar formula holds for X', Y' respectively. One has S(X) \ S(Y) = S(X') \ 
S(Y'). Moreover, since U(Y) = U{Y') and U(X) = U(X'), one has p Y = p Y > and 
Px — Px' ■ Thus formula (13 .7p follows from the following equality 

(3.8) F G (P(Y))=f G (V(Y% 

which we now prove. 

Recall that, by definition of interval reflection, the interval [v,w] is of the form 
v = ei > d\ > e2 > d 2 > . . . > e s > d s = w, where the e;'s are of the same type and 
the rfj's are of another type. Then S(Y) = {a} U S', where a = v — w = e± — d s 
and S' = {d t - ei+i}^ 1 . Since S' C IL(Y) one has J/3] = f3 for f3 G S'. Recall that 
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[a] is G-invariant and note that py is also G-invariant (since LT(Y) consists of odd 
roots). Therefore 

^))-r7N^( n>eJ ii- e -.) » 

Consider the simple arc diagram Z with the order d\ > > d,2 > ■ ■ ■ > d s > e\\ 
then S(Z) = S' U {— a}. Using Lemma [3.51 we obtain 

Observe that s = ht ^ +1 = ht ^~ w ) +1 anc i | a j = a—J^^s' P = Pz~Py- Summarizing, 
we obtain 



_ e Pz 



where the last equality follows from (ll.5p (because Z is a simple arc diagram). Since 
S(Y') C n(Y'), Theorem KWG gives also J" G (P(y')) = CR Y 'e"y'. 

Since II (Z) corresponds to the total order d\ > > d% > . . . > d s > e\ and 
n(y') corresponds to the total order e\ > d\ > e2 > di > ... > d s , one has 
U(Z) = r ei _ ds . . .r ei _ e2 r ei _ dl (n(r)). Using <^M> we get 

R Yie Py' = {-\f s - x R z e pz = -R z e pz . 

This establishes (13. 8p and completes the proof. □ 



To complete the proof of Theorem 11.11 we will need the following observation. 
Lemma 3.7. Formula (11.101) follows from (11.111) . 

Proof. Fix any set of positive roots A + and let II be the associated set of sim- 
ple roots. Take h G f) such that a(h) = if a G II D Ao and a(h) = 1 if 
a G II H Ai. Then a(h) = mod 2 if a G Ao and a(h) = 1 mod 2 if a 6 
Ai. We claim that F(e a ) = e *V=ia(h) e a changes (fTTTT]) to (fTTD]) . Let fc = 
e nV=i P {h)_ xhen, obviously, F(e p ) = A;e p , so we have F(e p R) = ke p R. We need 
only to check that F(e w ^) = k^0^e w(p) . First of all observe that F(w(e p R)) = 

sgn'(w)F(e p R) = sgn' [w)ke p R. On the other hand this equals F(e w( - p ^wR) = 
F(e w<yp} )F{wR). Since w permutes the roots of the same parity, we have that 
F(wR) = w(F(R)) and in turn F(e w ^)F(wR) = F(e w ^)w(F(R)). It follows that 
F(e w{ - p) wR) = e^^^^e^wiR) = e 7T ^ wM{h) sgn(w)e p R. Hence 

e *V=i»VKh) 8gn ( w } = ksgn'{w), 

and this relation implies our claim. □ 



We are finally ready to give the proof of Theorem 11.11 in the current setting, 
formula ( II. lip becomes 

(3.9) F WB (V(X))=C 5 e Px R x . 



By Lemma 13.41 any arc diagram can be transformed to a simple one by a sequence 
of odd and interval reflections. By Theorem KWG, JF Wi {V{X)) = C g Rxe Px if X 
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is a simple diagram. For any odd reflection r a of II one has p ra (u) = Pn + ot (cf. 
and so 

(3.10) e p ^R raU = -e^Rn. 

Since for a G S(X) one has V(r a (X)) = -V(X), the fraction F Wg (V(X)) /(R x e px ) 
is not changed by the action of r a . The interval reflections do not change R x e px , 
since they do not change U(X); moreover, applying Corollary 13.61 with W = W g , 
they do not change J 7 ^ (V(X)). This proves formula (I3.9p . hence fll.lip . Lemma 
13. 71 implies that (ll.lOp also holds. By choosing X to be a nice arc diagram, we have 
that [7] G Q + for any 7 G S(X). This concludes the proof of Theorem 11.11 

3.2.1. For future applications we will need a slightly stronger version of Theorem 
11.11 Given an arc diagram X, let B' be a subset of B containing Supp(X). Let 
A(jB') be the set of roots that are linear combinations of the simple roots that are 
in the span of B' . Assume for simplicity that A(B') is irreducible and let A^(B') 
be the irreducible component of Aq(B ; ) which is not the smallest one in the sense 
of |H Section 1.2]. Let W(B') and W^(B') be the corresponding Weyl groups. 
Clearly W B W\B') is a subgroup of W{&) and let T = (W B ,W\B'))\W (B') . Set 
Z = WJW{&') and W = ZW&W*(B') so that W s = W T. 

Proposition 3.8. 

(3.11) ^w (V(X)) = %^ Px Rx. 

Proof. Recall that any arc diagram can be transformed into a simple arc diagram 
by a sequence of odd and interval reflections. Note that these reflections permute 
the ends of arcs and do not change the positions of other vertices (the interval 
reflections do not change the order of vertices and the odd reflections permute two 
vertices connected by an arc). Thus these reflections do not change B' and Supp(X). 
Since WoWs U pp(X) — Wo, we can argue as in the proof of Theorem 11.11 and assume 
that X is simple. Let Y be the same arc diagram viewed as a diagram for A(B'). 
Then, since (p x — py, a) = for any simple root of A(B'), we have that p x — Py is 
iy(£>')-invariant. Since X is simple, Y is also simple and fll.5p gives 

FwKB>)(nY))) = ^wKb')t{V{Y)). 

It follows that 

F Wo (V{X)) = jF Wo/wKBI) ( e ox-o- . JF wK&) {V{Y))) 

= T^F Wo/ wHB>)( ePx ~ PY ■ ^wI{b')t{V{Y))) 

This completes the proof. □ 

3.2.2. Comments on type A. Note that if m 7^ n, formula fll.lip restricts plainly to 
A(m,n) = sl(m + l,n+ 1). If instead m — n, the formula does not restrict to i) 
when [7] = JZiS (#i — e i) f° r 7 S iS". Note that the factor 1 _ e 1 _ |[7] is W B -invariant, 
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hence it can be taken out of the sum. Since the left hand side of (11.111) restricts to 
f), the sum 

e p 

Fw *( Y[ (l-e-lflp 

I3es\{ 7 } 

is divisible by 1 — e - ^. After simplifying, we may restrict to the Cartan subalgebra 
of A(n,n) getting a superdenominator formula in this type too. 

3.3. Proof of Proposition 11.21 Recall from (11.81) the definition of ]7[; note that 
it may combinatorially rewritten as ]7[= Y2/3ey< sn/3 ■ sn7 ■ /3 and that ]7[= if 
7 G IT. Let 

e P+E 7es M 

Q(x) = IW 7 ^)' 

By Theorem KWG, we have ^F W #(Q(X)) = R x e px if X is a simple arc diagram. 
Notice that an odd reflection changes the sign of Rxe Px and does the same on 
Q(X). The interval reflections do not change Rxe Px , since they do not change 
U(X). Thus, due to Lemma 13.4} in order to prove (11.151) . it is enough to verify 
that the interval reflections do not change Fw{Q{X)). This is done in Lemma [3.91 
below. 

Lemma 3.9. Let X be an arc diagram and r\ V)W ] be an interval reflection. Then 

jW(QpT)) = T w #(Q{r [vM {X))). 

Proof. Denote by Y the arc subdiagram corresponding to the interval [v,w] and 
let Y' = r< v>w ](Y). Then X' = r[ V)W ](X) is obtained from X by substituting Y by 
Y'. View Y, Y' as arc diagrams of gl(k, fc)-type. Then G = W[ v>w ] D is 
constructed for Y and for Y' . 

Let e, d be vertices such that e — d G S(X). Denote by ]e, d[ the interval [e, d] \ 

{e,d}. Then ]e - d[= ±(E^] e ,d[n£^ ~ EceKwinuO with the si S n "+" if e G 8 
and the sign "— " if e G T>. We see that ]e — d[ is iy] ej( i[-invariant. In particular, 
]7[ is G-invariant for each 7 G S(X) \ S(Y), since if v',w' are vertices such that 
v' — w' G \ 5(F), then [u, 10] fl [u 7 , iw 7 ] = or [v, w] C [f', it) 7 ] (because the arcs 

do not intersect). Notice that (px — Py, ot) = for each a G n(F), hence px — Py 
is G-invariant. Therefore 

f PX-pY+J2~ teS (X)\S(Y)hl 

^#(Q(4-^#/ G ( n tt— — V MQ{Y))) 

VVp&S(X)\S{Y)\ L e J 

and the similar formula holds for X', Y' respectively. Notice that S(X) \ S(Y) = 
S(X') \ S(Y'). Since IL(Y) = IL(Y') and IL(X) = n(X'), one has p Y = p Y ,, 
Px = Px 1 - We see that the required equality J r w #(Q(X)) = ^' W /#(Q(X / )) follows 
from the equality 

(3.12) ? G (Q(Y)) = MQ(Y')), 

which we verify below. 

Recall that, by the definition of the interval reflection, the interval [v, w] is of the 
form v = t\ > d\ > ei > di > . . . > e s > d s = w, where the ej's are of the same type 



18 



GORELIK, KAC, MOSENEDER, PAPI 



and the dj's are of another type. Then S(Y) = {a} U S', where a = v — w = ei — d s 
and S' = {di — ej+i}^. Since S' C H(Y) one has 

s 

E M=W= X)(et - *) - «■ 

7£5(y) i=l 

Consider the simple arc diagram Z with the order d± > e 2 > d 2 > ■ ■ ■ > d s > ei, 
then S(Z) = S' U {-a} and 

1 s 

i=l 

We have 

* o(c(y » = ^IW^ 1 = '^ iwim ' = -^ (e(z)) - 

Since Z and F' are simple, one has J" G (Q(Z)) = e^-Rz and Jb(Q(Y')) = e^'R Y >. 
Arguing as in the proof of Corollary I3.6[ one shows that e pz Rz = —e PY 'Ry^ and 
this completes the proof of f)3.12p . □ 

Remark 3.1. Arguing as in Lemma l3.7[ one deduces (11.141) from (I1.15p . 



4. Distinguished sets of positive roots and compact dual pairs 

4.1. Dual pairs and Theta correspondence. Let us now recall what dual pairs 
and the Theta correspondence are: this involves some basic and well-known facts 
on the oscillator representations of symplectic groups (see e.g. [T] for more details 
and a rich list of references) . 

Let (V, (• , •)) be a 2n-dimensional real symplectic vector space. Fix a polarization 
V = A + © A~ {A ± are isotropic subspaces) and a standard symplectic basis w.r.t. 
(• , •): A + = ©™ =1 Me 4 , A- = e? =1 R/i, so that (e h f)) = 5 iy 

Starting from this polarization of V we can construct a complex polarization of 
Vc = V ®e C by setting 

n n 

Vc = C(e, + v^TA), V c = C(e, - >/=!/,). 
i=i i=\ 

This polarization is "totally complex", i.e. V,^ PI V — {0}. As in (I1.16P we can 
consider the representation M = W(Vc)/W(Vc)V^ of the Weyl algebra IV(Vc) 
of (Vc, (■ , -}c), and, by means of (11.171) . we define an action of sp(Vc, (■ , -)c), on 
M. This representation is usually called the oscillator representation. The choice 
of a totally complex polarization is equivalent to assigning a compatible complex 
structure J on V (i.e., J G Sp(V) such that J 2 = —1). Explicitly J is defined by 
setting J(ej) = —fa and J(fi) = e^. Let W be the space V seen as a complex space 
via the complex structure J. The elements g G Sp(V) commuting with J form a 
maximal compact subgroup K of Sp(V) and we let t be its complexified Lie algebra 
viewed as a subalgebra of sp(Vc, (• , -)c)- Since K commutes with J, we may let 
it act C-linearly on W . We let det(fc) be the determinant of the action of k G K 
on W . If K is the a/ det cover of K, then M has an action of K whose differential 
coincides with the action of t as a subalgebra of sp(Vc, (• , -)c)- 
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For future reference we describe explicitly this action. Recall that 

K = {(g, z) G K x C x | z 2 = det g}. 

The covering map is the projection it on the first factor. The Lie algebra of K is 
the subalgebra of t x C given by 

Note that dir is the projection on the first factor and provides an isomorphism 
between t and t. We want to define an action of K on M in such a way that 

(4.1) exp(X) ■ v = e d < x) (v) 

for any X G t. Identify V£ with W by mapping v <8> (a + ib) to av + J(bv). Then M 
is linearly isomorphic to the polynomial algebra P(W) on W by identifying v G 
with the linear function on given by v(u) = (u,v). Recall that K acts on W, 

hence also on P(W). With this identifications we can define an action of K on 
M ~ P(W) by 

(4.2) (g, z) ■ p = z~ l g ■ p. 

We now check that (14. ip holds. According to our definitions, if X = (A, ^p-) G I 

tr(A) * 

then exp(X) ■ p = e ~ e ■ p. On the other hand, according to (11.171) . A acts on 
M by left multiplication by 9(A). Now, applying (I1.18p . we see that 

9(A)p = [9(A), p] +p9(A) ■ 1 = A-p + p9(A) ■ 1. 

Choose a basis {xj} of and let {yi} be the basis of such that (xj, yj) = 5ij. 
Then {x*, yi\ is a basis of Vc and — X;} is its dual basis. Hence 

9(A) ■ 1 = -^A( yi )xi - 1 = - ^(x,-, A^))^ • 1 



Therefore 



--^(x t ,A(^)) = ^ 



0(A)p = A • p —p. 



Exponentiating, we find (14. ip . 

Let H be the element of sp(Vc, (■ , -)c) such that H\y± = ±1. Then bracketing 
with if defines a Z-gradation 

Sp(V C ,{-,-)c) = ^S P (Vc,{-,-)c)n. 

nSZ 

We set p = © n >o s p(Vc, (■ , -)c)n- Clearly p is a parabolic subalgebra of 
sp(Vc, (■ , -)c)- 

A reductive dual pair is a pair of real Lie subgroups Gi, G2 of Sp(V) which act 
reductively on V and such that each is the centralizer of the other in Sp(V). We say 
that the dual pair is compact if one of the two subgroup is compact. In the following 
we deal always with compact dual pairs, assuming G\ compact. We also assume 
that G\ C K and let Sj be the Lie algebras of Gi (i — 1, 2). Denote by sf , i = 1, 2 
their complexifications. Let G\ be the lift of G\ to K. Since G\ C K it follows that 
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the center of K is contained in G*2- The center of K is { exp(y/— ItH) \ t £ R}, thus 
we can conclude that H £ s^- It follows that p2 = p Hs^ is a parabolic subalgebra 
of S2 . Howe duality in this setting gives the following result (see also [15]): 

Theorem 4.1. [5] There is a set £ of irreducible finite- dimensional representations 
of Gi such that, as G\ x s^-module, 

M = Q) V ®T(rt), 

where t(t]) are irreducible quotients of p 2 -parabolic Verma modules. 
The map 77 H- r(?7) is called the Theta correspondence. 

We need also to recall the following structure theorem (cf. [7]). 
Proposition 4.2. 

(1) A compact dual pair (Gi,^) is of type I, i.e., G1G2 acts irreducibly on V. 

(2) A reductive dual pair (G\, G2) is of type I if and only if there exists a divi- 
sion algebra D over R with involution e, an Hermitian right D-vector space 
(W±, (-, -)i), a skew-Hermitian left D-vector space (W 2 , (-, O2) an d an iso- 
morphism V = W\ ®d W2 of "R-vector spaces such the symplectic form (-, •) 
corresponds to Ttd/r{{-, -)i ® e ("> O2) an( ^ under which G\ and G2 map to 
the isometry groups U(Gi, (■, U(G 2 , (-, -)2) ; respectively. 

4.2. Cartan involutions. Suppose that is a Lie superalgebra of basic classical 
type. If q is simple of type A(l, 1) let A be the set of roots of gl(2, 2). In all other 
cases we let A be, as usual, the set of roots of q. Choose a set A + of positive roots 
and let II = . . . , a n } be the corresponding set of simple roots. If g is not of type 
-4(1, 1) then the root spaces have dimension 1, so we can choose for each a £ A + 
root vectors X a £ Q a and X_ a £ g_ a with the property that (X a ,X_ a ) = 1, and 
let h a = [X a ,X_ a ]. We set e; = X a . and / 4 = X_ a .. If q is simple of type 4(1, 1), 
then, given a £ A, we let X a be the projection on g of the corresponding root 
vector in gl(2, 2). 

Recall from [9] that q is the minimal Z-graded Lie superalgebra with local part 

n n 

1=1 1=1 

and relations 

(4.3) [ei,fj] = Sijh ai , [h ai ,ej] = (a*, a,-)^-, [h ai , fj] = -(ai, ay)/j, 

on the local part. From now on we assume that (a^, atj) £ R for any In 
particular, if g is of type D(2, 1, a), we assume that a £ R. 

We let N a> p be the structure constants for the chosen basis of root vectors: 

[X a , X p ] = N a ^X a+p , if a, 0, a + £ A. 

Set a a = — 1 if a is an odd negative root and a a — 1 otherwise, so that (X a , = 
a a . We also let p(a) be the parity of a: p(a) = 1 if a is odd and p(a) = if a is 
even. The following statement is a reformulation of Lemma 3.2 of [S]. 

Lemma 4.3. Given a, £ A sitc/i t/iat a + /3 £ A, let p,q be non-negative integers 
such that (3 + £ A U {0}, iGZi/ and on/y i/ — p < i < q. 
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(1) If a is even, then N a ^N_ aja+/3 = \q{p + l)(a,a) 

(2) If a is odd and (a, a) ^ 0, then 



-a a \q{a, a) if p is even 
°~a\{p + ot) if pis odd 



(3) If a is odd and (a, a) = 0, then N a ^N_ a ^ a+ p 



cr a (a, j3) if p is even 
if p is odd 



Proof. The first statement follows as in the case when g is a Lie algebra. 

For the second statement, let us first assume that a and (3 are not proportional. 
Then, as in [SJ Lemma 3.2], we obtain 

v 

N a> pN- ata+ /3 = a a - ia){h a ). 

i=0 

Since 2a is an even root, we see that the subspaces 

-p<2i<q -P<2i+l<<j 

are the irreducible components of J2- P <i<q ^Qp+ia viewed as a (X 2a , h 2a , X-2 a )~ 
module. It follows that (3{h a ) + q(a, a) = —(3(h a ) + p(a, a) if p — q is even while, if 
q—p is odd, then (3(h a ) + q(a, a) = —f3(h a ) + (p—l)(a, a) and /3(h a ) + (q— l)(a, a) = 
— a). This implies that p— g is even and f3{h a ) = ^-(a,a). Substituting 
we find the statement. Suppose now that a and (3 are proportional. There are only 
two possibilities: = a or (3 = —2a. Both cases follow directly from the Jacobi 
identity 

[[x a ,X- a ],Xf,] = [x a , [x_ a ,Xp]] + {-iy^\x m Xpix_ a \. 

Finally, if (a, a) = 0, then, as in Lemma 3.2 of [8], we obtain 

v 

N a ^N_ a , a+p = a a 53(-l)*O0 - ia )( h a), 

i=0 

hence the statement follows readily. □ 
As shown in the proof Lemma 3.3 of [8], we have that 

Substituting in Lemma |4T3"| and using the fact that if a, f3, a + (3 e A then N a ^ ^ 0, 
we find 

(4.4) N a ^N_^_ a = -^-q(p + l)(a, a) 

if a is even, 



(T a <T 



(4.5) N^K 



2(7 



a+/3 



q(a, a) if p is even 



a,/3 iv -/3,-a \ -a a ap 



^(p + l)(a,a) if pis odd 



if a is odd and (a, a) ^ 0, and 

(4.6) N a ^N.^_ a = ZZaZlfap) 

0~a+l3 
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if a is odd and (a, a) = 0. 

Introduce the anti-involution T of q defined on the local part by 

'T(h) = h ifhet), 
T{e.i) = fi if (Xi is even, 

Tie-i) = y/— 1/j if oci is odd. 

Since it preserves relations (I4.3p . it can be extended to $j, and T(g a ) = As 
shown in Lemma 3.8 of [8], we can choose X a , X_ a in such a way that (X a , AT_ a ) = 1 

and T(X a ) = \/— l p a a a X- a , and we can still assume that X a . = and X_ a . = fi. 
It follows that, if a, j3, a + /3 6 A, then 

= [T(X i g),T(X a )] = A^./j _ Q ,v / -l _P( a a apX- a -p, 

hence 

(4.7) A^ = (-l^^AL,^ = -^AU,_,. 

Combining (J3T7]) with 033]), (Oj) . it follows that 

(4-8) iV^ = Ig(p +!)(«,«) 

if a; is even, 

f(-l)^^ if pis even 

1 yj ^"\-(-l)^)(H±i^) if pis odd 

if a is odd and (a, a) ^ 0, and 

(4.10) JV^ = -(-l)^(a,)9). 

if a is odd and (a, a) = 0. Thus N a >( g is either real or pure imaginary. 
If (a, a) ^ 0, let e a = sgn(a, a). Let 

if p(a) = 0, 



if p(a) = 1. 



Lemma 4.4. 

(4.11) AT^ = f^AV 

Proof. Since A^ = ±ATg >Q , it is enough consider the following three cases: 

(1) a is even; 

(2) a is odd non-isotropic and (3 is odd; 

(3) a and are isotropic. 

In case (1), by (14.81) . we have N a) p = £ a N ai p. Thus the result follows obviously if 
f3 is odd. If (3 is even, then observe that H4.8[) implies that N a >( g 7^ if and only if 
e« = C/3 = e a+/3- This observation implies the result. 

In case (2), by Lemma 14731 (2). the product N a ^N_ a ^ a+ p is real; by (1), N_ aa+ p 
is real iff ^ a+/3 = 1, so A^ is real iff £ a+/ 3 = 1 as required. 

In case (3), we have 2(a, (3) = (a + f3, a + (3), so, by (14.101) . N a ^ = ^ a+ pN a) p as 
desired. 
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□ 



In the above setting, given complex numbers Ai, . . . , A n such that Aj G V — 118. if 
p{o-i) = 1 and Aj G K. if p(ati) = 0, one can define an antilinear involution u : g — > g 
setting 

(4.12) u(e,i) = Xifi, u{f i ) = \J 1 e h uj{h ai ) = -h Qi , 1 < i < n. 

Since oj preserves relations ( 14. 3p . it follows that it extends to g. 

Clearly co(X a ) is a multiple of X_ a . We define X a for each a G A by 

(4.13) w(X a ) = -cr a £ a \ a X_ a . 

Hence, if a, (3, a + /3 G A, since w is an antilinear automorphism, we have 

~ &a+p£a+p^a+pN a ,p = O a O p£a€p^a^pN- a ,-p- 

Using ( ED and ( 14TTT]) , we deduce that 

(4.14) A a+/ 3 = AqA^, A a A_ Q = 1. 
It follows that, if a = YH=i n i a ii then 

(4.15) A a = n(--c.Ar- 

i 

Endow g with the Z-grading 

(4.16) s = 0qi 

which assigns degree to h G f) and to and /j if is even, and degree 1 to e, 
and degree —1 to if ctj is odd. 

Let 7r = {z | 1 < i < n,p(ai) = 1}, 7r c = {1, . . . , n} \ ir. 



Proposition 4.5. 

(1) The set of uj -fixed points in g Q is a real form of g . 

(2) qg is a compact form of qo if and only if Aj(aj, ai) < for all i G ti c . 

(3) // Aj(aj, «j) < /or all i G 7r c and \/ — l^i ^ fl,ye ^ e same sign for all i G n, 
then, for any positive integer r, (X a ,cu(X a ))(a,a) < if X a G q4 r © q_4 r 
and (X Q ,,a;(X Q ,))(a,a) > if X a G q 4r -2 © q~4r+2- 

Proof. The proof of the first assertion is standard. For the latter two claims observe 
that, by lj05]l . 

(4.17) (X aj w(X«)) = -a a e«A a . 

If a is an even root, the r.h.s. of (I4.17P is a real number whose sign does not 
depend on the choice of the basis of root vectors X a . In fact, if {X' a } is another 
basis with [X' a ,X'_ a ] = o- a h a , then X' a = c a X a for suitable complex numbers c a 
and (X' a ,cu(X' a )) = — \c a \ 2 a a £ a \ a . Therefore we can use formula (I4.15P to prove 
the statements in a straightforward way. As for (2), recall that a real Lie algebra is 
compact if the form (•, oo(-)) has signature opposite to that of the invariant form (•,•). 
If X a G q then X a = UhU^T 1 - Therefore -£ a a a \ a = -£ a UhU^T 1 > for 
any a if and only if Aj(aj,a!j) < for all i G n c . Claim (3) is proved in a similar 
way. □ 
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Let 

V = fixed point set of u\ Bl . 
Clearly V is a real form of Qi. Since a basis of V is given by 

(4.18) {X a + co(X a ) | X a E q i9 i E 2Z + 1}, 

it is checked easily that (• , •) = (-,-)|v is a real nondegenerate symplectic form. 
Since a<i| 01 ((go)iR) C ad\ gi (go) H sp(V), we have that 

arf| Sl ((g )») = ad| 9 i(flo) H sp(V) 
and that ad| 01 (go) H sp(V) is a real form of ad| 01 (g o ). 

4.3. Dynkin diagrams. We will briefly recall the usual encoding of sets of positive 
roots by means of Dynkin diagrams. In the following N = n + m + 1 and 0> ®> 
□ correspond respectively to even, isotropic and nonisotropic (both even and odd) 
roots. A • is a placeholder for even or isotropic roots. The possible diagrams in 
each type are listed in [9]. We reproduce this list below. 

4.3.1. Type gl(m + 1, n + 1). 

(4.19) ■ ■ ... 



a N a N+1 



4.3.2. Types B(m + 1, n + 1) and Dim + 1, n+ 1). The possible diagrams in these 
types are 

(4.20) • ■ . . . ■ =^ □ 

for type B[m + 1, n + 1), and 

(4.21) ... O^— O 



OjV <*N+1 



(4.22) • • . . . • ® O 

°1 Q 2 QJV-I «jv ajv+1 



(4.23) O 



a.N-1 



o 
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for D (m + 1, n + 1) . 

4.4. Distinguished sets of positive roots. 

Definition 4.1. We say that a set of positive roots A + is distinguished if qj = 
{0} for \i\ > 2 in grading (I4.16p . VFe say that a set of positive roots A + is very 
distinguished if the corresponding set of simple roots contains a unique odd root. 

Since maxja,,- | Y17=i a i a i e P( a j) = 1} = 2 for any choice of A + , we have 
that if A + is very distinguished, then it is distinguished. The sets of positive roots 
which in [TO] and in most of the subsequent literature are called distinguished are 
all "very distinguished". We have preferred to change the terminology because of 
the relationships with the Theta correspondence. 

We now discuss the possible distinguished subsets of positive roots up to In- 
equivalence. The classification is basically a case by case inspection, looking at all 
possible diagrams (cf. [9] and [22] for the exceptional types) and calculating the 
coefficients which express the positive roots as a linear combination of the simple 
roots. 

4.4.1. gl(m,n). Given non negative integers p, q with p + q = m, we let A^'^ be 
the set of positive roots for gl(m,n), corresponding to the following set of simple 
roots 

~^g{ qS} = {ei — £2, • • • , , £p ~ Si, 5i — 62, ■ ■ ■ , S n — e p+ i, e p+ i — e p+ 2, • • • , e m _i — e m }. 

This is essentially the only possibility for a distinguished set of positive roots (in- 
cluding the possibility p = or q = 0, in which case q» = {0} for \i\ > 1). The 
other one is to take non negative integers r, s such that r + s = n and exchanging 
e's with 5's. But this case is equivalent to the above by exchanging m and n, so we 
will not distinguish these two possibilities. 

4.4.2. B(m,n). There is a unique, up to W^-action, distinguished set of positive 
roots A~g. The corresponding set of simple roots, with notation as in [9], is 

(4.25) ITb = {61 — 6 2 , . . . , 6 n — ex, e x — e 2 , . . . , e m _i - e m , e m }. 

We also allow the case m = 0, in which the set of simple roots is {5± — 82, ■ ■ ■ , 5 n -i — 
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4.4.3. D(m, n) . There are three distinguished sets of positive roots, up to iy g -action 
: Aj 1; Aj 2 , Aj 2 ,. The corresponding sets of simple roots are 



U D1 


= {81 


-62,. 


■ ■,S n 


— €1, €1 — €2, ■ ■ ■ , e m _i — e m , e m _i + e m }, 






- £2, • • 




-1 — ftrn e m — 81, S\ — 62, ■ ■ ■ , <5 n -l — ^n; 25 n }, 


n_D2' 




- £2, • ■ 




-1 + Cm, —Cm ~ <^1, Si — 62, ■ ■ ■ , S n -i — S n , 25 n } 



AAA. C{n + 1). There are three distinguished sets of positive roots, up to In- 
action: Aq 1: A£ 2 , A^ 2 ,. The corresponding sets of simple roots are 

IIci = {Si-5 2 ,..., 5 n _i - 5 n , 5 n - e, 5 n + e}, 

11(72 = {e - #i - $2, ■ ■ ■ , S n -i - &n, 25 n }, 

nc2' = {-e - #1, 5i - 6 2 , • • • , <5„_i - £n, 25 n }- 



4.4.5. Exceptional types. A direct inspection shows that the distinguished sets of 
positive roots correspond to the following diagrams 



(4.26) 




for _D(2,1, a) (left display; it will be denoted by AJ, 21q A G(3) (right display; it 
will be denoted by A^), and to 



(4.27) 



- Q < P - 



-O 



- Q < ( ) 



for F(4) (they will be denoted by Aj 1; A^ 2 , respectively). 

4.5. Distinguished sets of positive roots and real forms. If A + is distin- 
guished we choose uj corresponding to \ = —e ai if a, is an even simple root and 
Aj = y/—T if ati is an odd simple root. Set, for any a e A+, 



-{X a — y/—iX- a ), f a — — ^=(X_ 



■ix a ). 



Since 0i = qi + q_i, applying formula (I4.18P with our choice of uj, we get that these 
vectors form a standard symplectic basis of V. Since 

C(e Q ± >/=T/ a ) = fl f , 

aeA+ 

the oscillator representation is exactly M A+ (qi) as a sp($ji)-module. 

Observe now that for any Lie superalgebra of basic classical type we have 

(4.28) 0o = 0o x 0o 
with Qq, reductive Lie algebras. Set 

(4.29) Si = ad lsi { d i)nsp{V), z = l,2. 
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A case by case check in the distinguished sets of positive roots shows that we can 
always choose gj, $Jq so that: 

So c <Jo, 0o = (qongg)©(q_2 + q 2 )- 

In the case of g = gl(m,n), we choose gj = gl(m), q% = gl(n) or the other way 
around. Combining these observations with Lemma 14.51 we readily obtain the fol- 
lowing result. 

Proposition 4.6. If A + is any distinguished set of positive roots of a Lie super- 
algebra of basic classical type, then Si is a real form of ad\ Sl (g ) . Moreover s i is a 
compact form of ad\ Sl (Ql) and there is a Cartan decomposition t ffit of s 2 such that 
t c = ad\ Sl (gl n q ) and r c = ad| gi (q 2 © q_ 2 )- 

In particular, if q 2 = q_ 2 — {0} then Si x s 2 is a compact form of ad\ Sl (q ) = 
ad| gi (g ). 

4.6. Detour: classification of real forms. We now deepen some aspects of the 
theory developed in the last two Subsections, in order to obtain an explicit con- 
struction of all real forms of Lie superalgebras of basic classical type, which have 
been classified in [9], [18] and [21]. In this section when referring to type A(m,n) 
we mean either g = gl(m,n) or the simple superalgebra of type A(m,n). 

Let gnj be a real form of g and let u be the corresponding complex conjugation. 
Then a;| go is an antilinear involution of the Lie algebra go, hence there is a corre- 
sponding Cartan decomposition g = t © p, with Cartan involution We will 
describe explicitly the involution ( w . 

Let K be a compact Lie group having i w as Lie algebra and let be the Lie 
algebra of a torus in K Q . Then t = tn ©r C is a Cartan subalgebra of 6. Let 1) 
be the centralizer in go of t. It is clear that both t and f) are w-stable. Since t) is 
(^-stable, we have f) = (f) fl t) ffi (f) fl p); since ojC,^ = (^u, we have u(t) D p) C rj fl p. 
So h D p = (h D p)^ + y/^T(i) n p) w . It is known that t R ffi y/-L(1) fl p) w is a Cartan 
subalgebra of a compact real form of go; therefore roots are purely imaginary- valued 
on it. So the real span of roots is contained in a/— Hr ffi (f) n p) w . Now fix a set of 
positive roots A + and let uq be the involution given by f)4.12p with the following 
choice of parameters: 

(4.30) Aj = — e ai for i n, Aj = \/— 1 for i E n. 

Then 

(uou )\i = I, {u ocj )|ftnp = —I- 
In particular, t is a Cartan subalgebra of Qq ouj ° . It follows from [ITJ Chapter 8] that 
(u o coo)\ So = ?7o o e^C 1 ) with r] a diagram automorphism of go and h E t. Clearly 
r] = uj o cu o e - ad ( h ) i s an extension of 7/0 to g hence we can write that 

u = n o e ad(h) o w , 

with 77 an automorphism of g such that ?7| go is a diagram automorphism of go- We 
now list the possible choices for automorphisms 77 of go, such that ?7| go is a diagram 
automorphism of go- 

Lemma 4.7. There exists rj E Aut(o) such that rj\ so is a nontrivial diagram auto- 
morphism if and only if g is of type A(m,n) , D(m,n), D(2,l,a), aE {l,(—2) ±1 }, 
and r]\ So is as follows. 
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(1) If g is of type A(n,m) with n ^ m, then r]\ go restricts to the nontrivial 
diagram automorphism of both A n and A m . 

(2) If g is of type A(n,n), then r]\ Bo is either the nontrivial diagram automor- 
phism of both A n components, or it is the flip automorphism between the two 
A n components, or the composition of these two automorphisms. 

(3) If g is of type D(m, n), m > 2, then t]\ Q0 is the unique diagram automorphism 
of go- 

(4) If g is of type D(2,l) = D(2,l,a), a e {1, (-2) ±1 }, then r] is the unique 
diagram automorphism of the diagram in the left display of ( I4.26P . 

Proof. No 7] as in the statement can exist if go has no nontrivial diagram automor- 
phisms. This rules out the cases F(A), G(3), B(m, n) with (m,n) ^ (2,2). The 
case when g is of type 5(2, 2) is also excluded because the flip automorphism 
of g = B 2 x B 2 sends the unique odd simple root to a weight which is not a 
root. Let us now deal with type A(m,n). An automorphism of g of type A(m,n) 
which restricts to the nontrivial diagram automorphisms of both A n and A m is 

given by ( ^ ^ ^ — ( ^ ^ t J . The flip automorphism F is given by 

F ( C D ) = ( BA ) md the ^position Fo(- si ) gives the remaining an- 
tomorphism. We are left with proving that there is no automorphism of g of type 
A(m, n) which restricts trivially to, say, A n and nontrivially to A m or it is obtained 
from this one by composing it with the flip. This is easily checked by observing 
that these latter automorphisms map the simple odd root to a weight that is not a 
root. 

If g is of type D(m,n), an automorphism that restricts to the unique dia- 
gram automorphism of go is Ad(J) with J — ( ^ m ~ 1 > 1 ® j ; where I m -i,i = 



Im-l 

-1 



e gl(m). 



We now deal with the case when g is of type D{2, 1, a). If a G {1, (— 2) ±1 }, then g 
is isomorphic to D(2, 1) and claim Ad(J) gives as above the desired automorphism. 
We need to check that, if a £ {1, (— 2) ±1 }, then there is no automorphism rj of g 
which restricts to a non trivial diagram automorphism of go and that, if a = 1, then 
Ad(J)\ go is the only one. We check this as follows. Recall that g = (s/(2,C)) x3 , 
and let (3i,/3 2 ,(3 3 be the simple roots of these copies of s/(2,C) Since the form 
(v(')yV(')) 1S invariant, then there exists a constant c such that (rj(-), rj(-)) = c(-, •). 
If r] restricts to a diagram automorphism of g , then rj(j3i) = (3 a (i), i = 1,2,3, for 
a suitable permutation a. Hence {P a {i)i P<?{i)) — C_1 (A) A)- We check directly that 
this is not possible for a real value of a different from 1, (— 2) and also that, if 
a = 1, then the only possibility is given by Ad(J). □ 



The case of g being the simple Lie superalgebra of type A(l,l) needs special care, 
so we postpone the discussion of this case to the end of this section. Until then, we 
are excluding this case from our discussion. 

In order to complete the classification we choose a very distinguished set of posi- 
tive roots. If g is of type D(m,n), we choose Aj x . We wish to compute the action 
of oj on the generators e*, /j. We will often use the following result of Serganova. 
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Let 5\ be the map on g defined by setting 

(4.31) h\ So = Id, S Xlst = \Id, 6 X{a - = X^Id. 

If g is of type J, then 5\ is an automorphism of g for any A G C. If g is of type II, 
then 5\ is an automorphism of g if and only if A = ±1. By [2"U| Lemmas 1 and 2], 
an automorphism of g that restricts to Id go is necessarily 5\. 

Since the form (t)(-),t)(-)) is invariant, we have that there is s v G C such that 
(t)(-),t](-)) = s v (-, ■). Since rj 2 = Id on g , we see that = ±1. Let now i be the 
index of the unique odd simple root. Set 7 = 77(0^). Then there is A G C such that 
v( e io) = ^X-y so that rj(fi ) = s r? a 7 A _1 X_ 7 . If r]\ go = Id\ so , then we set 771 = Id g . If 
rj\ go 7^ Id\ go and g is of type A{m, n), then we set 771 = 77 o 5 A -i, while, if g is of type 
D(m, n) we set 771 = Ad( J) with J as in the proof of Lemma T4. 71 Then, in all cases 
Vi( e io) = -^7 an d we can write 

u = Vl 5 x e ad( - h W 

We want to find conditions on h and A for which has order 2. 

First assume that r7| go = Id. In this case fj = t and 77 = 5\ for some A G C. 
We see that u(e lQ ) = y^lX'^o W/^, ufa) = -e^^fr if z ^ z , cj(/ io ) = 
\/— lAe Qi o ( h ^ej , cu(ej) = —^ ai ^~ ai<yK) fi if i 7^ Since a; is an involution we obtain 
that Ae^oW e R and e a < (/l) G R if z 7^ z , hence setting A io = V^TA^e -0 ^^ and 
Aj = —e ai e~ ai ^ we see that u; has the form of (14.121) . 

Assume now that 77| go 7^ Id. Recall that 77 acts on g as a diagram automorphism. 
Since, by our choice of the set of positive roots, the simple roots of go are simple in 
g if g is of type A(m,n) while, in type D(m,n), rji is a diagram automorphism of 
g, rji induces in both cases a transposition i \-t i' on the indices corresponding to 
the simple even roots. If i <ji ir, then 

Hence 

Since h G t we have oti'{h) = oti(h), therefore we get e *^ G R. 
We distinguish the following cases. 

(1) If g is of type D{m,n), then 77(0;^) = a io and s m = 1, hence 

uj 2 (e i0 ) = e a ^ h) e- a 'o^e io . 

Therefore e Ql o (/l) G R. 

(2) If g is of type A(m, n) and (?7i)| 0o = (— st)\ go , then <r 7 = —1 and (r]i) = 
(—st)5^ for some \i. Hence (771) 2 = (— st) 2 , and we see that (vi)f Sl — —Id. 
Moreover s vi — 1. 

It follows that 

u 2 (e t0 ) = \\r 2 e^e- a 'oWe lo . 

Since h G t, we have 7(/i) = ai (h), hence |A| = 1 and e a *oW g R. 

(3) If g is of type A(n, n) and (771) | Bo is the flip, then (?7i)| go = F\ go . It follows 
that o 1 — — 1 and 771 = for some /x Hence (?7i) 2 = F 2 , so ( r 7i)f Sl = 
Moreover s m = —1. 

It follows that 

u 2 (e i0 ) = |A|-VWe-<*oCO eio . 
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Since h G t, we have j(h) = ai (h), hence |A| = 1 and e a *oW e R. 
(4) If g is of type A(n,n) and (?7i)| go = F(—st) then 7 = a io and s m = —1. 
It follows that 



hence Ae ai oW e ^/^lR. 
Let us now return to the general case. Fix h as above and define 

Wa = ^ fMh) UQ in cases ( 2 ), (3) 

<V r TA ead(/l)w o in case (4). 

As observed above Uh is an antilinear involution of the form of (I4.12p . 

We are now ready to give an explicit formula for the Cartan involution corre- 
sponding to u)\ go . Define the map £ Uh : g — > g by 

(4.32) L h (x) = x, xet), C h (X a ) = sgn(X h a )X a , 

where A^ is the multiplier defined in (I4.13p . It is easy to check using (I4.14p that 
( UJh is an involution of g . We claim that 

is the Cartan involution corresponding to u\ go . 

First we prove that sgn(X^) = sgn(X^^) for a G Aq\ We prove in fact that 
A^ = X^ a y A direct computation shows that = c a e~ a ^ for any a G Ag , 

where c a = < „ a ^ c i a j m follows since our choice of h forces 

I -A^ 2 ifX a Gq ±2 

a(h) = rji(a)(h). 

Clearly rji (X a ) = d a X m ^ a y We now prove that \d a \ = 1 for any a G A . For this 
observe that d- a = s m d~ l and d m ( a ) = d~ l . Hence, since u 2 (X a ) = (r]iUJh) 2 {X a ) = 
X a , it follows that 

1 \h \h A A 

so 1 = e a e^ m ( a \s m \d a \~ 2 = \d a \~ 2 as wished. 

In order to prove that is the Cartan involution corresponding to OJ\ So , we need 
only to check that 00 and £ w commute and that u\ So o £ w is the conjugation of a 
compact real form of Qq. The first claim is a simple computation: 

^\soC^( x <y) = -e r)1 ( a )S5f?7,(A^)A^ l(Q , ) (i Q (i„ r;i ( a )A_ Q , = -s Vl e Vl ( a )sgn(Xa)X a X- a , 

while 

(ujU\ 30 (X a ) = —e a sgn(X h _ Vi ^ a - ) )X l ^d_ a d_ ril ( Ka - ) X- a = — e Q s5fn(A^)A^X_ Q , 

so the claim follows from the observation that s Vl e r]1 ( ,\ = e a . 

For the second claim it is enough to check that e a (X a , u)£ u (X a )) < for any 
a G A . Indeed 



e a (X a ,u( u (X a )) = -e a sgn(X a )X vl(a) e m(a) s Vl = -sgn(X a )X a , 

and this number is always negative. 

Recall that we are excluding the case when g is the simple Lie superalgebra 
of type A(l,l). It follows from [91 Prop. 5.3.2] that, if cu and to' are antilinear 
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involutions of g such that $Jq is isomorphic to 0q , then to' is conjugated by an 
isomorphism of g to either u or to o 8_ x . Consider now the antilinear involution uj^ 
on g defined by u R (ei) = e,, u R (fi) = fc, and u R (h) = h for he yf-tif . Then, 
since Uh is an antilinear involution of the type described in f)4.12p . we have that 
u R u h u R = U) h 5-t. Moreover, if 771(01) C g+, we have that = ijiUfc, so u R 

provides a real isomorphism between g w and g^ 5-1 . If instead ?7i(g 1 ") C g]~, then 
g is of type A(m, n), and 5 v /tt provides an isomorphism between g w and g' 1 " 5-1 . It 
follows that we need only to classify the symmetric pairs (go, t) up to isomorphism. 
First of all we consider cases (3) and (4). In both cases (Cw)| g i gives an isomorphism 
from qI — > Qq, so the map (X, Y) h-> X + Cu{Y) gives an isomorphism between 
qI x qI and go that maps the diagonal copy of gj to the set of fixed points of 
Thus the real form q$ is isomorphic to gj seen as a real Lie algebra. 

Let us now turn to the other cases. Choose t G f) in such a way that e ai ® = 
sgn{\^) for i 7^ «o- We claim that we can choose t G t. Since sgn(A^.) = 
s 9 n (^ 1 (a i ))^ ft * s c l ear that we can choose t so that ?7i(aj)(t) = aj(t) for i 7^ i . It 
remains to check that we can choose t so that oii {t) = r)i(a>i )(t). This is clear if 
Vi( a i ) = a iai so we are om y w ith case (3). In this case we know that 

e a io (h) _ e Vi(ai )(h) _ e -a iQ (h) J~J e ~ a i( h ) _ 

This implies that (e a ^ h) ) 2 = e~ a ^ h) . It follows that e~ a ^ h) > so that 

e a i0 (t) = e -a i0 (t) "Q sgn ( e -<*iW) = e m(* i0 )(t) 

hence we can choose t G t as wished. 

If g is not of type A(m,n), then the simple roots of g are linearly independent, 
so we can further assume that e Qi o(') g R. If g is of type A(m,n) and 7/1 = Id s , 
then we can choose A so that Ae" 4 "^ G IBL If, instead, we are in case (3), then, as 
shown above, (e Qi oM) 2 = 1 ; so, again, e ai o^ G M. This implies that we can define 
an antilinear involution Observe that ( u = Cu' so u and uJ define 

the same real form. Moreover = r\\ e ad, ^ t •* with t = t' if g is of type C(n + 1) or 
A(n, m) and t' defined by setting a^t) = aj(t') for z 7^ i , cti {t') = |a/— 1~ + a io (t) 
in the other cases. 

We will now use Kac classification of Lie algebra involutions [TT] to classify £ w . 
Let {(3{, . . . ,(3 l k } be the simple roots of (g^ 1 corresponding to the set of positive 
roots Aj[. Recall that /3* = a\ t for some simple root of g corresponding to Aq. Let 

X l N be the diagram of g and let (X l N )^ be the corresponding affinization with 
Ti = 1 if (771)^ = -fd and rj = 2 otherwise. Let a* be the labels of (X^)^ and set 
ftj = J2j>i a jPj- Using the analysis done in [121 §3] we find that there is r G t and 
w in the Weyl group of gj} 1 such that 2n\/^lt" = w(t') + r with e ad ^ = 5^ and 
t" G t fl [go, go] having the following property: if = /?](£") and s = ^ — 8i(t"), 
then 

(4.33) s} G {0, -, 1} and ^ ajsj = -. 
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It follows that e^v^^X*") = e ^(0 f or a n i ^ i Q an d that \^ e ^ Q )(^V^it") = 
Xe ai o^'\ This implies that e^v-i^iC*") ^ f or a vj { ^ Z ' Q Moreover, in type A(m, n) 
and C(n+ 1) with 771 = Jc/g, A/i^e^o^v 7 " 1 *") e ffi. In type A(m, n) with 771 as 
in case (3), since r £ t we must have /x 2 = 1 so e ai o^ 27T ^^- t " s> £ IR. In the other 
cases we have e 27r v /r Tai ( i ") ^ ^/— In turn, this implies that there is an antilinear 
involution uj" such that = rjie 27T ^^ ad ^"\ Since £ w » is conjugated to C, we see 
that u and a;" define isomorphic real forms. The outcome is the following: 

(1) If g is of type C{n + 1) or A(m,n) and r/i = Id g , then the list of all t" £ 
!} n [fl 0> flo] that satisfy f H~3"3"j) and such that there is A £ C with Ae^ot 27 ^ 1 *") £ M 
gives a list of all real forms. Since the latter is an empty condition, we see that the 
list of all real forms is given by all the t" £ f) n [go, go] that satisfy (I4.33p . 

(2) If g is of type A(m, n) and 771 is as in case (3), then the list of all t" £ tfl [go, go] 
that satisfy (I4.33P and such that e 27r ^/-^ a i ( t ") ^ ]R gives a list of all real forms. 

(3) In all the remaining cases the list of all t" £ tfl [go, go] that satisfy (14.331) and 
such that e 27rv/rTai o(*") e y/— IR gives the list of all real forms. 

The set of t" £ tfl [go, go] that satisfy (14.331) is finite, so we need only to check 
the above conditions on this finite set. The outcome of this computation is given 
in the following tables. In all cases except A(m,n) and C{n + 1) define £ fj by 
aj{wi) = 5ij where the simple roots of g are enumerated from left to right. If g 
is of type A(m,n) or C(n + 1) we define Wi for i 7^ i to be the unique element 
in f) n [go, go] such that ai(zuj) = 5ij for all i 7^ i . In the following tables we 
assume that A + is a very distinguished set of positive roots. For types D,C,F we 
choose Apj, Aj 2 , Aj 1; respectively. Table I displays the real forms corresponding 
to antilinear involutions with 77| 0O = Id So and Table II covers the remaining cases. 
In Table I we also list the parameters (Ai, . . . , A n ) occurring in (I4.12p . setting, only 
once in the paper, i = \J — 1; the Xj which are not listed are equal to 1. 
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real form of go 


t" 


(Ai, . . . , A„) 


gl(m,n) 




u(p, m — p) x u(q, n — 


q) 


(0 < p < m - 1, 1 < q < n) 


A m = i 

Xj = A m + g — ~ 1 

(1 < j < m, j ^ p) 


A(m — 1, n - 


-1), 


su(p, m — p) x sit(g, n 


-q) x R 




same as above; 


m 7^ n 








(0<p<m-l,l<g<n) 




A(n — 1, n — 


1) 


sw(p, n — p) x su(q, n 


-<7) 


(0 < p < n - 1,1 < q < n) 


same as above 
with m = n; 


B(0,n) 




sp(2n,R) 






A„ = i 

A, = -1 

(1 < j < n - 1) 


B(m, n) 




so(2m + 1) x sp(2n,R 


) 




A n = i 

An+fc = — 1 
(1 < fc < m) 


B(m, n) 




so(2p, 2m + 1 - 2p) x 


sp(2n, R) 


(1 < P < m) 


A„ = — i 

A„+fe = — 1 

(1 < k < m, k ^p) 


D(m, n) 




so(2m) x sp(2n,R) 






A n = i 

An+fe = — 1 
(1 < fc < m) 


D(rn, n) 




so(2p, 2m - 2p) x sp(2n, R) 




A„ = — i 










(1 < p < m - 2) 


An+fe = — 1 

(1 < k < m, k ± p) 


D(m, ri) 




sp(n) x so* (2m) 






A„ = i 

An+fc = — 1 

(1 < fc < m- 1) 


_D(m, n) 




sp(q, n — q) X so* (2m] 




(1 < q <n - 1) 


A„ = — i 

Aq = A ?l + fc = — 1 

(1 < k < m - 1) 


C(n + 1) 




sp(n) x R 







Ai = i 


C(n + 1) 




sp(2n,R) x R 






Ai = i, A n +i = — 1 


C(n + 1) 




sp(g, n — g) x R 




iroq+i, 1 < g < n — 1 


Ai = i, \ q = —1 


D(2,l,a) 




(sZ(2,R)) x3 






H,i,i) 


D(2,l,a) 




sm(2) x su(2) x sl(2,l 






(»,-!,-!) 


F(4) 




so(7) x sl(2,R) 






(t, -1,-1,-1) 


F(4) 




su(2) x so(l,6) 




2^2 — 4^1 


(t, 1,-1,-1) 


F(4) 




su(2) x so(2,5) 






(-i, -1,1,-1) 


F(4) 




so(3,4) x sl(2,R) 






H, -1,-1,1) 


G(3) 




Q 2 ,o x sZ(2,R) 






G(3) 




g 2 , 2 x sZ(2,R) 




— + |n73 


H,i) 



Table I 



s 


^100 


real form of go 


t" 


(7/(m, n) 




-st 


gl(m,R)) x gl(n,R 


) 







A(m, n),m 


7^ n 


-st 


sl(m,R)) x sl(n,R 


) x R 







A(n, n) 




-st 


sl(m,R)) x sl(n,R 


) 







gl(2m,2n) 




-st 


u*{2m)) x w*(2n) 




K^m. + 
o^m. + 
2 W m + 


^ ro 2m+n 
I_ 

^^m+n 


A(2m,2ri), 


m ^ n 


-st 


su*(2m)) x su*(2n] 


x R 


A(2n,2n) 




-st 


su*(2m)) x su*(2n] 




gl(n,n) 




F 


gl{nX) 









A(n, n) 




F 


sl{n, C) 









D(m, n) 




Ad(J) 


sp(2n, M) x so(2m 


-2p- l,2p+l) 




2^n+p 












(0 < p < m - 2) 



Table II 
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In Table II, we have not considered the case when g is of type D(2, 1, — |) with non 
trivial r)i, because D(2, 1, — |) is isomorphic to D(2, 1). The corresponding real form 
is, from Table II, sp(2,R) x so(3,l). The special isomorphism so(3, 1) = s/(2,C) 
allows to recover the classification as stated in [TSJ Theorem 2.5]. 

Suppose now that g is simple of type A(l, 1). It needs special care since in this 
case there is a large group of automorphisms acting identically on the even part 
(this was apparently missed in previous discussions on real forms in the literature). 
First observe that go — sl(2, C) x sl(2, C), hence there are four non isomorphic real 
forms of g : s/(2, C), sl(2, R) x sl(2, R), su(2) x s/(2, R), su(2) x su(2). All these 
real forms are obtained from real forms of g by restricting an antilinear involution 
u of gl(2, 2) to sl(2, 2). Since this restriction clearly stabilizes the center of sl(2, 2), 
it can be pushed down to A(l, 1). Hence for any real form of go there is at least one 
real form of A(l, 1) which induces it. The only non-trivial thing to prove is that 
also in this case the real form of go determines the real form of g. Suppose that u 
and {J are antilinear involutions that restrict in the same way to go- Then there is 
an automorphism g of g such that g\ go = Id 30 and u' = cog. We now identify the 
group S of automorphisms of g that restrict to the identity of go with SL(2,C): 
indeed, if g is such an automorphism, then necessarily 

(4.34) g{X a2 ) = aX a2 + bX„ e , g{X- a2 ) = cX e + dX_ a2 

(4.35) g(X e ) = aX e + 6X_ Q2 , g(X_ e ) = cX a2 + dX_ e . 

The fact that g is an homomorphism implies that M g = ( ° ^ ^ G SX(2, C). 

Hence we have a map to SL(2, C). To prove that this map is bijective, we consider 
the local Lie superalgebra G_i © G © d with d = CX ai © CX a2 © CX a3 © CX_ e , 
G_i = CX_ ai © CX_ Q2 © CI_a 3 © CXe, Go — f) and whose bracket is obtained 
by restricting the bracket of A(l,l). The corresponding minimal Z-graded Lie 
superalgebra is g. A direct check shows that the map g defined by (14. 34 p . (I4.35P 
and such that g\ go = Id defines an automorphism of the local algebra, hence it 
extends uniquely to an automorphism of g. 

Suppose now that to = r]ie ad ^tUQ. It is easy to see that for any g G S there exists 
a unique g' G S such that 

M g > = AW g A, 

with A = ( J J J if flgf = su(2) x su(2) or sl(2, R) x sl(2, R), with A = 

if g £ = s l(2, R) x su(2), and with ^=(^J J^ if So= s/ ( 2 > C )- If w ' = then 
the condition that (w') 2 = Id s is equivalent to u = gug, hence we must have 

AWgAM^^l 

Moreover we have that [g'Y^ujgg' = ug" with M g » = AM~} AM g M g < . 

Consider the antilinear antiinvolution a on SL(2, C) defined by a(M) = AM A. 
We have just shown that the set of M g such that (u o g) 2 = Id is the fixed point set 
S of a. Let us also consider the action of SL(2, C) on S by M ■ s = a(M)sM. If 
this action has a unique orbit then, as shown above, oj and u o g are conjugated, so 
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the real forms g 1 ^ and g w ' are isomorphic. To check when this action has a unique 
orbit we observe that the stabilizer of any point s G S is the fixed point set of the 
antilinear involution on SL(2, C) given by (7 1 — s~ 1 a(g)~ 1 s hence it is a real form 
of SL(2, C). In particular all the orbits have the same dimension so the orbits are 
the connected components of S. More explicitly 

S = {( X _ Z J G SX(2,C) I x,y G R, z G C} 



if gft = su{2) x sm(2) or $ = sZ(2,R) x s/(2,R), while 

5 = {{ 4- ] G SX(2,C) I x,y G R, z G C} 



if 0q = sZ(2, C). In both cases S is the quadric in R 4 of equation xy 
where z = a + \/—lb, which is homemorphic to R x S 2 . 

It remains to deal with the last case, i.e. g$ = s/(2, R) x su(2), where 

S = { ( X Z )e SL(2, C) I x, y G R, z G C}, 



a 



2 



z y 

which is the quadric of equation xy — a 2 — b 2 = 1. This is homeomorphic to M 3 x 5°, 
hence disconnected. However, the two orbits are the orbit of Mjd and of Ms_ 1 (cf. 
f)4.3ip ). so we need only to check if g^ and g^ 5 - 1 are isomorphic, but the argument 
used in the other cases shows that cur provides an isomorphism between g w and 
gw<5_i Thus, A(l, 1) has no real forms other than those listed in Tables I and II. 

Remark 4.1. Our treatment doesn't cover the cases when the Cartan matrix is not 
real, which happens only if g is of type D(2, 1, a), where a G C\R, , a + a = —1. In 
this case one has one extra real form, with even part sl(2, C) x sl(2, R) (see |18j). 

Remark 4.2. Our approach to classify real forms started by fixing a suitable set of 
positive roots and henceforth the antilinear involution uq. All antilinear involutions 
are then gotten as rjie^^uo, by letting h and 771 vary. 

One can, in a different perspective, consider just the antilinear involutions rjiUJo 
and let the choice of A + vary. In this way one can associate to any set of positive 
roots a set of real forms indexed by the outer automorphism r)\. It can be checked 
that all real forms can be obtained in this way. The real forms discussed in Section 
14.51 are precisely those corresponding to the distinguished sets of positive roots with 
rji = Id g . In Section T4.8I we will discuss the real forms corresponding to the sets 
of positive roots of depth 4 when g is gl(m,n) or of type B(m,n) and D(m,n). It 
will actually turn out that, in these cases, all real forms of g already appear as real 
forms corresponding to positive sets of roots of depth at most 4. 

Remark 4.3. It is possible to deal with the problem of classifying antilinear in- 
volutions in a slightly different way. Consider marked sets of positive roots. For 
this we mean the datum (II, L) where II = {ai, . . . , a n } is the set of simple roots 
of a set A + of positive roots and L = {/j, . . . , l n } is a set of labels satisfying 
li G {±1} if p(oti) = 0, k G {±\/^T} if p(cti) = 1. Given a marked set of pos- 
itive roots, we can define an antilinear involution ujl using formulas (j4.12p -( l4.15p 
with Aj = —£aik> so that X ai = We have a natural action of odd reflections 
on marked set of positive roots: if r ai denotes the odd reflection w.r.t. the simple 
root a t (see fl£5)), we set r ai (H,L) = (r Qi (n),L'), with V = . . . ,l' n ], I) = l 3 
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if (aii, Oj) = 0, i 7^ j, l\ = Zj~ and E = Ijli if (ctj, a-,) 7^ 0. This action has the 
following property: if r a (U,L) = (H',L'), then, since £ UL = . , ojl and u;^/ define 
isomorphic real forms. Using this action, one has a combinatorial recipe to detect 
the real form corresponding to a given marked set of positive roots: since any two 
sets of positive roots can be obtained from each other through a sequence of odd 
reflections, it suffices to consider the marked sets (Jl v d, L v d), where U v d is one of 
the sets of simple roots corresponding to a very distinguished set of positive roots 
we used for the classification. Given any marked set (II, L) one can compute com- 
binatorially the marked set (U v d, L v d) in the same equivalence class and apply our 
classification to w^. 

4.7. Compact dual pairs coming from distinguished sets of positive roots. 

Let g = gl(m, n) or a Lie superalgebra of type B,C,D. Fix a distinguished set A + 
of positive roots. Let oj be as in Subsection 14.51 and let V be the set of cu-fixed 
points in Q±. Also recall from (I4.29|) the definition of Si,s 2 . 
In this Subsection we prove the following proposition. 

Proposition 4.8. Let g be gl(m,n) or a Lie superalgebra of type B,C,D. If A + 
is a distinguished set of positive roots, then there is a compact dual pair {G\i G2) in 
Sp(V, (• , •)) with Lie(Gi) = Si, % = 1, 2, such that the action of go on M A+ (qi) gives 
the Theta correspondence for G 2 ) at the level of Lie algebras. The compact dual 
pairs (Gi,G 2 ) are listed in the following table (in which m,n are positive integers): 



(4.36) 






A+ 


(Gi, G2) 


gl(m, n) 




(U(n),U(p,q)) 


B(0,n) 




(0(l),Sp(2n,R)) 


B(m, n) 


AS 


(0(2m+l),Sp(2n,R)) 


D(m, n) 


A^ 


lol2m),Sp(2n,R)) 


D(m, n) 


A£ 2 


(Sp(n),SO*(2m)) 


D(m, n) 


A + 


(Sp{n),SO*(2m)) 


C(n+1) 


A^ 


(0(2),Sp(2n,R)) 


C{n+1) 


A^ 2 


(Sp(n),SO*(2)) 


C(n+1) 


A + 


(Sp(n),SO*(2)) 



We shall prove Proposition 14.81 by realizing explicitly the superalgebras as sub- 
algebras of some gl(r,s) and then checking the conditions of Proposition 14.21 in a 
case by case fashion. For shortness we will give the details only in type D(m,n). 

4.7.1. A pi . Endow the superspace C 2m ' 2n with the bilinear form given by the matrix 

I ), where ( _l '•). 

The Lie superalgebra osp(2m,2n) of type D(m,n) is the set of linear transfor- 
mations which are skewsupersymmetric w.r.t. a supersymmetric nondegenerate 
bilinear form. If we choose the form as above, then osp(2m, 2n) can be realized as 
the set of matrices 

A B 1 B 2 \ 
—B\ C\ C 2 
B\ C 3 -C\ ) 



DENOMINATOR IDENTITIES FOR LIE SUPERALGEBRAS 37 

with A skewsymmetric 2m x 2m and C 2 , C3 symmetric n x n matrices. Denoting 
by Eij the matrix units, set 



f ~AEi tTn+ i + y/—lE m+ i t i, 1 < i < m, 

— lE2m+i2m+n+i + \/ — 1 ^2m+n+i,2m+i > m+l<2<m + n. 



Then f) = ©™+ n C/ij is a Cartan subalgebra of g. 
Set (X,Y) = \str{XY)] then 



(hi,hj 



5y 1 < i < m, 

m + l<z<m + n. 



In the chosen distinguished set of positive roots we have = {5i ± €j | 1 < j < 
m , 1 < i < w}. We can choose for each a G root vectors X a , X_ a in such a 
way that 

y/-LA 
The map * : 0i -> M 2 ™ 2n (C) 



intertwines the adjoint action of g on g x with its action on M 2m ,2n( ( C) given by 
(4.38) (V ° f Ya = EA-AF. 

It is then clear that ad ^ ^ (V) C if and only if 2£ and F are real matrices, 

i. e. E e so(2m) and F e sp(2n,M). 

Consider now the map $ : V — > R 2m (g) (M 2n )* given by 

$( V /Z T(^,2m+i + ^m+n+j.i)) = e; <g> / j < 1 < 2m, 1 < j < U 

$( V /Z T(^,2 m +n+i - E2m +j , t ) = d ® f H+j < i < 2m, 1 < j < U 

where {e^} is the standard basis of R 2m and is the standard basis of (M 2n )*. 
Note that $ intertwines the action of ad\ Sl (g ) D sp(V) on V with the standard 
action of so(2m) x sp(2n, R) on R 2m ® (R 2n )*. 

Let (■ , ■)2m be the standard symmetric bilinear form on R 2n , (■ , -) 2n the standard 
symplectic form on (M 2n )* and let (• , ■) be the tensor product (• , -) 2m g) (• , -) 2n . It 
is easy to check that, for j, k e {0, . . . , 2m}, i, r G {1, . . . , n} 

(Ej t 2m+i + E2m+n+i,j, E^^m+r + -^2m+n+r,fc) = 0, 
(Ej^m+n+i ~ E2rn+n+i,ji ^,2m+n+r + ^2m+r,fc) = 0, 
(Ej,2m+i + E2m+n+i,j, Ek^m+n+r ~ E2 m +r,k) = Sjk^ir- 

It follows that ($(X),$(y)) = for any X,Y E V. According to the clas- 

sification given in § 5], the pair (0(2m), Sp(2n, R)) is a type I dual pair in 
5*p(M 2m g) (IR 2 ™)*, (■ , •)). Since $ maps ad\ Sl (g ) H sp(V) C sp(V) exactly on the Lie 
algebras of this dual pair, we have proven Proposition 14.81 in this case. 
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4.7.2. A J 2 . Endow the superspace C 2m ' 2n with the bilinear form given by the matrix 



l m,m 

J. 





2d 



, where 



Ira 
Ira 

With this choice of the form, the superalgebra osp(2m, 2n) can be realized as the 
set of matrices 

/ A 1 A 2 B x B 2 \ 
A 3 - t A 1 B 3 B A 
t B 4: t B 2 C\ C 2 

- tS 3 — D\ C-3 — Oi J 

with A 2 ,A 3 skewsymmetric m x m and C 2 ,C 3 symmetric n x n matrices. In this 
case we set 



V 



hi 




E r a+i,m+i i 
— E„ 



Then f) 



1 < % < m, 

'■'m+i,ra+i ^m+n+iyra+n+ii ~t~ 1 — % — ^ 

)^ n C/ij is a Cartan subalgebra of q and 

1 < i < m, 
m + l<i<m 



(hi, h 



j ■ 



'J 



n. 



In the chosen distinguished set of positive roots we have = ± 5j | 1 < i < 
m, 1 < j < n}. We can choose for each a E Af root vectors X a , X_ a in such a 
way that 



(4.39) 
V = 





-hn'A 



A 




A 



-lA 2 -i 



.4, 



-1A, 



Ai, A 2 e M„ 



Again, the map given by f)4.37p . intertwines the adjoint action of go on Qi with 

( E 

its action on M 2rrit2n (C) , given by ( I4.38p . It is then clear that ad 



if and only if 
(4.40) 



F 



(v)cv 



E 



[ A1 - 


A 2 \ 


■ M 


B 1 


B 2 \ 


1 ~A 2 


Ax ) 




-B 2 


Bi ) 



with Ai,A 2 G M mjm (C), Bi,B 2 e M nn (C), A±,Bi skew-Hermitian, A 2 antisym- 
metric, and B 2 symmetric. 

Let H be the skew field of real quaternions. Set (HP)* = End^W 1 , M) where 
HP, HI are viewed as left H-spaces. Endow (H n )* with a right H-action by setting 



(Xq)(v) = X(v)q. Identify V with (H n ) 
followed by left multiplication by L m = 
of M 2mt2n (C) of matrices of the form 



H m as follows. The map \l/ above 




"-tlr, 



maps V into the subspace 



(4.41) 



A -B 
B A 



Identifying H r with C x C = C 2r by x + yj -H- (x, y) we see that the matrices 
as in (14.411) are precisely those commuting with the left action of j. Thus V gets 
identified with EndmiW 1 , H m ) (here HP, H m are seen as left H-spaces). The natural 
map A®w^ T\ tV with T x , v (u) = X(u)v identifies End m (M n ,U m ) and (HP)* <g> e H m 
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Let now $ : V — > (H n )*®eH m be the (M-linear) isomorphism described above. Note 
that, if E, F are as in gaO]), then both L m EL~l and F are of the form (jPIjl . Thus 
we can identify them with elements T E ,T F in EndaiET, H m ) and £nd H (H n , H n ) 

respectively. Unwinding all the identifications we see that if X = ^ ^, j G go 

and ad(X)(V) C V then $ad(X)$ _1 acts on (H n )* <g> e H m via the natural action 
of T F ® I + 1 <g> T E . 

Let r : H — > H be the quaternionic conjugation: r(a + 6j) = a — bj (a, 5 G C). 
It induces the complex conjugation in the above identification W = C 2r . Let 
(- , -)m be the skew-Hermitian form on H m given by (v,w)' m = 'Y J i v i^f~~^ T ( w i) 
and (■ , •)„ the Hermitian form on W 1 given by (v,w) n = J2i v i T ( w i)- Let (■ , ■)„ 
denote also the form induced on (H n )*. Note that, since E, F are as in ( I4.40p . then 
T E G u(U m , (■ , -)'J ~ so*(2m) and T F G u((U n )*, ( , )„) ~ sp(n). 

Let (■ , ■) be the real symplectic bilinear form on (H n )* £g> HT m given by 

(-,■) = Tr H/R (- , •)„ ® r o (• , -) m . 
For 1 < i < m, 1 < j < n, set 



(4.42) X tj = ^(V^lEij - E m+i>n+j ), 

(4.43) Yy = V-^Eij - y/^lE m+i>n+j ), 

(4.44) Z l3 = q-\V^lE itn+j + E m+iJ ), 

(4.45) Wij = M> ; (/•-',.„ . ; + V^T^+ij)- 



If X, F G {Xy, Fy, Zjj, H 7 ^}, then it is easy to check that (X, Y) = except in the 
following cases: 

2 = (Xij,Yij) = —{Yij,Xij) = (Zij,Wij) = —(Wij,Zij). 

On the other hand, letting {e^} be the canonical basis of H m and {e 1 } the basis 
dual to the canonical basis of H™, we see that 



(4.46) $(Ay) = e J ® v^e,, 

(4.47) $(^ i ) = e 3 '®e ij 

(4.48) $(%) = e J ® y^Ijei, 

(4.49) $(Wi i ) = -e J '®je i . 



It follows that ($(X), $(F)) = -2(X, F) for any X, F G V. 
According to the classification given in [5j § 5], the pair 

([/((IF)*, (• , •)«), U(M m , (• , .) m )) = OSfc(n), SO*(2m)) 

is a Type I dual pair in Sp((W n )* <g) HP 1 , (• , •))• Since the map <3> maps ad\ Bl (go) PI 
C sp(V) exactly on the Lie algebras of this dual pair, we have proven Propo- 
sition 14.81 in this case. 

4.7.3. Aj 2 ,. This case can be reduced to the A^ 2 follows: by a careful 

choice of the root vectors, the space V is the set of matrices 

( A ) 

{ -J 2n 'A ) 
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such that 



.4 



V 



Ai 


A 2 


V 


w 


V=1A 2 


-V=1A! 


—\f—\w 





with A t ,A 2 e M m _i >n (C), v,w E C n . Define S m>2m G M 2m>2m (C) as 



m,2m 



' Im-1 








\ 











1 













V 


1 





0/ 



Let a : g — » g be the map 

—■hnB C 



I — > 



Sm..2m.AS. 



m,2m 



■J2nBS 

m,2m 



Sm,2mB 

c 



It is easy to check that a is an automorphism of g. Since <x(V) is the space described 
in (I4.39P we see that the map X h-^ $ o arf(cr(X)) o $ -1 identifies ad\ 3l (g ) fl sp(V) 
with sp(n) x so*(2m). Finally observe that str(a(A)a(B)) = str(AB) hence 

($Ki)),$HF))) = -2(i,y) 



for any X, Y G V. This concludes the proof of Proposition 14.81 in this case. 

4.8. Noncompact dual pairs and gradings of depth 4. In this subsection 
we classify, up to PV s -action, the sets of positive roots such that the grading (I4.16P 
has depth at most 4 (i.e., q« = {0} for |i| > 4) and we show that these sets of positive 
roots are related to the noncompact dual pairs (U(p, q), U (r, s)), (0(p, q), Sp(2n, R)), 
(Sp(p,q), SO*{2n)). These pairs exhaust the type I dual pairs, with the exception 
of the pair (0(m, C), Sp(2n, C)). The latter pair cannot arise in our picture, since 
0(m, C) x Sp(2n, C) is not a real form of go for any choice of g. It occurs as the 
real form g fl g^ with g = osp(m, 2n) x osp(m, 2n) and g« the diagonal copy of 
osp(m, 2n) in g. Since we are chiefly interested in the application to dual pairs, we 
confine to study gl(m, n), B(m, n), D(m, n), C(n + 1). 

In the following we choose u (see f l4.30p ) as antilinear involution and let A° be 
the corresponding multipliers as defined in f l4.13p . Recall that the corresponding 
involution £ Wo of g is defined by setting (^(Xa) = sgn(X^)X a . 

Case g = gl(m,n). It is clear that diagrams of type A with three or four grey 
nodes support sets of positive roots whose associated grading has depth 3 or 4, 
respectively. These sets of positive roots correspond to the following sets of simple 
roots: 



(4.50) {ei — e 2 , . . . , e p — Si, 5% — 6 2 , . . . , 5 r — e p+ ±, e p+ i — e p+2 , . . . , 

6p + l — 5 r +l, <5 r +l — &r+2i ■ ■ ■ i $r+s~l ~ ^r+s}) 

(4.51) {ei — 62, .., 6h — Si, S\ — S 2 , ..,S r — €h+l, €h+l — Cfi+2, ■■, £h+q ~ S r +1, 

S r +1 ~ S r +2, ••) — S r+S , S r+S — €h+q+l, ■■, £h+k+q-l ~ e h+k+q] 

and are clearly the only ones with the required property about the grading, up to 
switching the role of e and S. Here p + q = m, r + s = n in f)4.50p and h + k = 
p,p + q = m,r + s = nm (14.5 ip . Fix f)4.5ip as a set of simple roots. To detect 
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the real form corresponding to cu , we need only to classify (Cu )\ s p i = 1, 2 via the 
parameters s*-, % — 1, 2, 1 < j < fcj, , fci = m — 1, ki = n— 1 as explained in Section 
14.61 In this case the roots are the simple roots of q 1 : 

P) = otj, l<j<h - 1, /9? = 1 < j < r - 1, 

/3ft = + &h+l + • • • + Q!h+r, fir = &h+r • • • + aft+r+g, 

= a h+r+j , 1 < j < q - 1, = aft +r+(?+i , 1 < j < s - 1, 

= Q>h+r+q + • • • + Olh+r+q+s, 
Ph+q+j = a h+r+q+s+j, 1 < J < & — 1. 

while 0j is the highest root of 0q. In this case £ Wo = e 2,rv/rT * with /3j(t) = | if 
sgn(\°pi) = —1 and = otherwise. Note that, letting = (3j(t) and s l = 

1 — then t satisfies (I4.33p . It follows that, since = for j 7^ h,h + q, 

s h = s h+q = |, s 2 = for z 7^ 0, r, Sq = s 2 = 2> the corresponding real form of Q is 
u(p, q) x tt(r, s). 

Case g = B(m,n). Recall that in type B all diagrams have shape (14.20]) . Assume 
I vi" I = 2. Suppose first a n+m G n: then a n+m is odd and non isotropic. The grading 
has depth 4. The simple roots of go that are not simple in q are both of degree 
2, while the largest roots are of degree 4. Arguing as for gl(m,n) it is easy to see 
that the corresponding real form is so(2m, 1) x sp(2n,M). The corresponding dual 
pair is (0(2m, 1), Sp(2n, M)). Otherwise the set of positive roots is necessarily of 
the form 

{ei — 62, • • • , e P — Si, 61 — 82, ■ ■ ■ , 5 n — e p+ i, . . . , e m -\ — £m, e™} 

with p + q = m. This grading has depth 4, since the highest root is ei + 62- Arguing 
as above, we can show that it gives rise to the dual pair {0{2p, 2q + 1), Sp{2n, K)). 
It is easily seen that if \ir\ > 3 then the grading has depth strictly greater than 4. 

Case g = D(m,n). If the diagram of the set of positive roots is like (I4.21|) . then 
arguing as in the previous case we deduce that the only sets of positive roots with 
I vi" I = 2 and depth at most 4 (indeed exactly 4) are 

{Si — 62, ■ ■ ■ , 5 P — €1, €1 — e 2 , . . . , e m _i — e m , e rn — S p+ i, . . . , 8 p+g -i — 5 p+g , 25 p+q }, 
{Si — S2, ■ ■ ■ , S p — ei, €1 — e%, . . . , e m -i + e m , —e m — S p+ i, . . . , S p + q -i — S p+q , 2S p+q ). 

with p + q = n, which gives rise to the dual pair (Sp(2p, 2q), SO* (2m)). If instead 
the diagram is like (I4.23p . we again obtain a unique set of positive roots: 

(4.52) {ei — e 2 , . . . , e p — 5\, 5\ — 82, ■ ■ ■ , S n — e p +i, • • • , e m -i — e m ; ^m-i + e m } 

with p + q = m, which gives rise to the dual pair (0(2p, 2q), Sp(2n, M.)). In both 
cases, if |vr| > 3 then the grading has depth strictly greater than 4. 

The pairs (0(2p + 1, 2q — 1), Sp(2n, R)) are gotten by considering the antilinear 
involution uj = 7]iUq when the positive sets of roots are those in f)4.52p and rji = 
Ad(J). We need to classify = r)i C^ - The extended Dynkin diagrams of 0q,0o 
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are of type Dm\ C n respectively. The corresponding roots /3* are 
P) = oij, 1 < j < p - 1, P* = a p+j , 1 < j < n - 1, 

/3 p = «p + + • • • + a p +n, /?2n = 2(a p + n . . . + a m+n _2) + a m + n _i + a m+n , 
Pl+j = a P+n+j, 1 < j < q - 2, 

The parameters for are s* = | and s] = for j ^ p, s 2 n = | and s| = for 
j n. It follows that the real forms of g l are respectively so(2p + 1, 2q — 1) and 
sp(2n, R), which give rise to the dual pairs {0(2p + 1, 2q — 1), Sp(2n, R)). 

Note that the pair (0(1, 2m — 1), Sp{2n, E)) is obtained from the distinguished 
set of positive roots whose set of simple roots is 

(4.53) {5i — 5 2 , ■ ■ ■ , S n — €i, €i — € 2 , ■ ■ ■ , C m -1 — ^m; £m-l + e m}- 

Case q = C{n + 1). According to p[ page 52], the only sets of simple roots which 
correspond to non-distinguished sets of positive roots are 

{5i - S 2 , ■ ■ ■ , Si - ei, ei - S i+1 , S i+1 - S i+2 , 25 n }, 

{5i - 5 2 , ■ ■ ■ ,5i + e 1 ,-e 1 - 6 i+1 , S i+1 - 5 i+2 , • • • , 25 n }. 

where i ranges from 1 to n — 1. The associated gradings have depth 4. 

5. Theta correspondence for the pair (0(2m + 1), Sp(2n, M)) 

In this section we use the denominator identity developed in the previous sections 
to derive the Theta correspondence for the compact dual pair (0(2m+l), Sp(n, R.)). 
This pair, according to Proposition I4.8[ corresponds to the distinguished set of 
positive roots A^ in a superalgebra g of type B(m,n). Recall that 

(5.1) A = ±{ei ± £j , e u 6 k ±5 l ,26 k \l<i^j<m,l<k^l< n}, 
Ai = ±{5 k ± 6i, 5 k I 1 < % < m, 1 < k < n}; 

(5.2) A = ±{e i ±e j ,e i ,6 k ±6 l \l<i^j<m, l<k^l<n}, 

Ai = ±{5 k ± €i I 1 < % < m, 1 < k < n}. 

Also recall that the defect of g is d — min(n,m). In this Section we will use the 
following notation 



(5.3) 


A(B r ) 


= ±{ei ± €j, €i | 1 < i ^ j < r}, 


(5.4) 


A(CV) 


= ±{4 ± 5 h 25 k | n - r + 1 < k ^ I < n}, 


(5.5) 


A(A r _0 


= ±{5 k -5i\n-r + l<k^l<n} 1 


(5.6) 


W(A-i) 


= Weyl group of A(yl r _ 1 ), 


(5.7) 


W{B r ) 


= Weyl group of A(B r ), 


(5.8) 


W r 


= subgroup of W s generated by i = 


(5.9) 


W r + 


= {w eW r \ w = s 2 s n ■ ■ ■ s 2 5 ik , k even}, 


(5.10) 


W r ~ 


= {w e W r 1 W = S 2 8 n ■ ■ ■ S 2 5 lk , k odd}. 



Now fix the set of positive roots A+ (cf. (ET25]) ). Then 2p 1 = (2m+l)(5 1 +. . .+S n ). 
Set A + (C„) = A+ n A(C n ), A + (B m ) = A+ n A(B m ), and A + (A n ^) = A+ n 
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A(A n _i). We denote by p c , p B , p A the corresponding p- vectors. With notation as 
in Section 13. 1\ the total order corresponding to this choice of A + is 

61 > ■ ■ ■ > 5 n > ei > ■ • • > e m . 

There is only one arc diagram associated to this total order, namely 

X = {5 n €i, . . . , 5 n -d+i€d}- 
The corresponding maximal isotropic set of roots is S(X) = {71, . . . , 7^}, where 

(5.11) 71 = 5 n - e 1; 72 = 5 n _i - e 2 , . . . ,7^ = ~ e d- 

We apply Proposition 13.81 with B' = Supp(X). Note that, with notation as in 13.2. 1| 
A(Supp{x)) = A(B d ) x A(C d ), and we can choose A*(Supp{X)) = A(B d ), so that 
W Su pp(x)W i {Supp{X)) = WiAa-tjWiBd). With notation as in lETS) . set 

^ = ^(A n _!)>V n _^(B m ). 

We can choose 

Z = (W(A n _ 1 )/W(A d _i))>V n _ ( i(W(5 rn ,)/l^ r (5 d )), 
so that W = W. Since nf=i fe ^ 7 ^ +1 = d\, Proposition 13.81 gives 

(5.12) e p R = jF w {—. ). 

mi(i-e- M r 

Dividing (1512} by £> = e p0 FLeA+C 1 ~ e ~")' we find 

(5.13) c/iM A+ ( 01 ) = = - = -Lf w (— ). 

Let F d = {a = (a 1; . . . , a d ) G (Z + ) d | ai > a 2 > • • ■ > a d } the set of partitions 
with at most d parts. Recall from Section |4]that there is an element H G f) such that 
a(if) = 1 for all a G A^(g), thus the domain f)+ = {/i 6 f) | a(/i) > OVa G A^} is 
nonempty. Since, by our choice of VF, in the denominators of ( 15. 13ft only sums of 
roots in —A± occur, we can expand the r.h.s. of (I5.13p in a product of geometric 
series on thus obtaining the following equality of formal power series: 

(5.14) chM A+ ( Sl ) = V ±-T w (e^-Eti ■**) . 

a 6 P d U 

We can write V = e"° FL^W 1 ~ e - ")^ FL e A+(£ m) (l - e~ a ). For a G F d , 
define 

d m 

p(&) = -^2(a d+1 ^ r )5 n - d+r , e(a) = ^a r e r . 

r=l r=l 

Using fl5.14p . we obtain that 

chM A+ ( fll ) = E 



aeP d \u,eW(A„_i)Wn-d ri Qe A+(c n )( 1 

E e t«(p s +e(a))-p s 
sgn(u;) — -, 

,weW(B m ) llaeA+cs™)! 1 -e 
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Here we used the fact that sgn(w) = sgn'(w) if w G W(B m ). By the Weyl character 
formula, we see that 

e w(p B +e( a ))-p B 

2_, s 9n(w)— — — - = chF B (e(a)), 

w & W(B m ) HaeA+(B m )l i e J 

Fb(e(8l)) being the finite-dimensional irreducible so(n)-module with highest weight 
e(a). We therefore obtain that 

e w(- Pl +p(a)+p c )-p c 

(5.15) chM A+ ( Ql ) = Yl E s 9n'(w)= _ chF B (e(a)). 

aeP d «,eW(A„_ 1 )w„_ d llaeA+(C„)V J 

If A G span(5i), set 



(5.16) chF A (X) = 



sgn{w) 



As the notation is suggesting, when A + p" 4 is regular and dominant integral for 
A(-4 n _i), then c/zFa(A) is the character of a finite-dimensional irreducible represen- 
tation of sl(n, C), while, if A + p A is singular, then c!iFa(\) = 0. 

Using the fact that sgn(w) — 1 if w G W n _d, we can rewrite (15 .15)) as 

(5.17) <AM- + ( Bl ) = E E ^^^f^c ft F B ( £ (a)). 

Recall from the previous section that we constructed a real symplectic subspace 
V of 0i and a map $ : 1/ -> R 2m+1 <g> (IR 2 ™)* such that there is a dual pair (G 1; G 2 ) 
in sp(V, ( , )) having the properties described in Proposition 14.81 We now want 
to describe explicitly the set E and the map r occurring in Theorem 14.11 In what 
follows we show that all these information can be read off from formula (j5.17p . 

First of all we need to parametrize the finite-dimensional irreducible representa- 
tions of G\. To accomplish this we start by describing G\. With notation as in 
the previous section, let J 1 : (M 2n )* — > (IR 2 ™)* be the complex structure such that 
J'(ft) — —fn+i and J'(f n+ i) — fi for i — 1, . . . , n. A direct computation shows 
that, if J is the compatible complex structure on V introduced in the previous 
section, then $J$ _1 = Id® J'. It follows that, if we let W be the space (M 2n )* 
seen as a complex space via the complex structure J', then, obviously, $ is a C- 
linear isomorphism between W and the complex space IR 2m+1 ® W. We look upon 
0(2m + 1) as a subgroup of Sp(M. 2m+1 Cg> (M 2n )*) via its action on the first factor. 
Then G x = | g G 0(2m + 1)}. It follows that, if Q^g® G G x , then 

det w ($~ l g®) = det R 2m+i® w ,(g) = (det R 2 m +i(g)) n . 

Therefore 

{($- 1 5 ,$ ) ±i) | g e 0(2m + 1)} if n is even, 

G 1= { {($"V$,±1) \g£SO(2m+l)} . 

, — , , if n is odd. 

U{($^$, ±V^1) | -g G SO(2m + 1)} 

Let G\ ~ SO (2m + 1) be the connected component of the identity of G\. From the 
description of G\ we see that G\ = G\x Z with Z isomorphic to Z/2Z x Z/2Z if n is 
even and isomorphic to Z/4Z if n is odd. The generators of Z are (—Id, 1), (/rf, — 1) 
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in the first case, while Z is generated by (—Id, \J— 1) in the second case. It follows 
that the finite-dimensional irreducible representations of G\ are pairs (F, x) where 
F is a finite-dimensional irreducible representation of G\ (hence of its complexified 
Lie algebra ) and x is a character of Z. 

If n is even and e G {1, —1}, we let Xe be the character of Z such that Xe{—Id, 1) = 
e and Xe{Id, — 1) = — 1. If n is odd we define Xe to be the character of Z such that 
Xe(~Id, v^T) = -V-Le- If a G P d set e(a) = (-l)£ii a *. 

In our case the set of roots of 5% is A(C n ). In the identification of f) with f)* 
given by ( , ), we see that the element G f) that corresponds to — YH=i ^ nas 
the property that -H|y± = ±-f- Thus the parabolic subalgebra p2 defined by if is 

P2 = f) © ^ (flo)a © n, 

a6A(A n _i) 

where n = ^ (flo)a is the nilradical. 

aeA+(C„)\A(A n -i) 

Notation 5.1. If X E (f) D S2)* is such that X + p A is regular and integral for 
A(v4 n „i), we let L 2 (X) be the irreducible quotient of the p2-parabolic Verma module 
V 2 (X)forsl 

If fi G (f) Hs^)* is such that fi is regular for A(A n _i), we let {p} be the unique 
element in the W(A n _i)-orbit of p that is dominant with respect to A + (A n _i) . 
Finally, we set c M = sgn{v), where v is the unique element ofW(A n _i) such that 

= M- 

Note that 

chV (A) = c x+pA - (i-e-Y 

Proposition 5.1. With notation as in Theorem \4-l\ we have that, if (F,x) E S, 
then there exists aePj and e G {1,-1} such that 

F = F fl (e(a)), x = Xe- 

Furthermore, the ^-character of the isotypic component of (Fb(e(sl)), x±e(a)) i n 
M A+ ( 01 ) is 

/mm chF A {w(p c - pi + p(a)) - p c ) 

(5.18) ^ -= (i- f -a) chF B«a))- 



fW^L d are defined in fl£D, (ISTTUj) ). 

Proo/. If (F, x) occurs in M A+ (gi) then, by ( 14~TI) . F = F B (e(a)) for some a G P d . As 
in the previous section, we identify M A+ (fli) with P(W). Let P(I¥) + and P{W)~ 
be the subspaces of homogeneous polynomials of even and odd degree respectively. 
By the explicit expression for the action of K given in (14. 2p . we see that Z acts by 
Xi on P{W) + and by X-i 011 P{W)~ . This proves the first assertion. 

Let M(a,Xe) be the isotypic component of (F B (e(a)),Xe)- By Theorem 14. 1} if 
M(a, X e)^{0}, then 

chM{a, X e) = chL 2 (X)chF B (e(a)) 

for some A G (f) fl s 2 )*. 
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Observe that H acts on a homogeneous polynomial p as H ■ p = (— pi(iJ) — 
deg(p))p. Since M(a, Xc ) = L 2 (\) <g> F B (e(a)), we see that we must have 

e = (_ 1 )(A+e(a)+Pi)(^) > 

It is well known that chL 2 (\) = Cx^chV 2 (p) , where A — p = X] a eA(c„) n " a - 
Since a(H) is even for any root in A(C n ) we deduce that 

(5.19) chM{a, X e) = cx^chV 2 (p)chF B (e(a)) 

p 

with e = (-l)(^+ £ ( a )+pi)( ff ). 
Hence, if we set 

(5.20) W T n Z = {w £ W n - d | w(- Pl + //(a) + p c ) is regular for A(4»-i)}, 
we derive from (I5.17P that 

(5.21) c/iM(a, xi) + c/iM(a, x-i) = 

^ ^^^({^(-p! + //(a) + p c )} - p c )c/iF B (e(a)), 

where c M = c w( _ Pl+Al(a)+p c). 
We need to compute 

(_l)(M~Pi+Ma)+P C )}-p C +£(a)+pi)CfO_ 

For this observe that e(a)(H) = and that u>(p(a)) = p(a) for all a £ and 
w £ W n -d, hence 

(w(- Pl +p(a) + p c ) - p c + e(a) + pi)(#) 

= p(a)(F) + (-w( Pl ) + + (^(p c ) - p c )(H) 

= p( a ){H) + {-w{ Pl ) + Pl )(H) mod 2. 

Since (-l)M a )W = e ( a ), Pl - s 2S .(p 1 ) = (2m + l)5. h and v(H) = H for any v £ 
V^(yl n _i), we see that 

(_]_)({' u, (-pi+A t ( a )+p c }-p c + £ ( a )+pi)(- H ') = ±e(a) if w £ >V ± d . 

If w,w' £ W„_d with w ^ w' then W(A n _i)t<; D W(A n -i)w' = 0, hence the set of 
characters {chV 2 ({w(— p\ + /i(a) + p c )} — p c ) \ w £ W^!f d } is linearly independent. 
It follows that, if Af(F B (e(a)),x e (a)) 7^ {0}, then, comparing with flCTj) . we 

find 

c^M(F fl (e(a)),x e («)) 

c ro c/iy 2 ({w(-pi + p(a) + p c )} - P c )chF B (e(a)). 

Likewise, if M(F B (e(a)), x_ e(a) ) ^ {0}, then 
cfcM(F B (e(a)),x_ e(a) ) 

^ c w chV 2 ({w(- Pl + p(a.) + p c )}-p c )chF B (e(a)). 

Since c/iF^A) = if A + p A is singular, (15.181) follows. 
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Finally if M(Fb(e(bl)), X±e(*)) = {0}, then the linear independence of the char- 
acters involved implies that VV^!f d D W^L d = 0, hence (15.181) holds as well. □ 

Let P* denote the set of all partitions with exactly j parts and set 

m 

(5.22) P= y 

j'=max(0,m+l 

For max(0, m + 1 — (n — d)) < j < m and a G P*, we set 



3 ' 

j'=max(0,m+l— (n— d)) 



n—j n 

(5.23) i/(a) = - 2^ ^ ~ a «-H-A- 

r=n— d— m+j r=n—j+l 

Corollary 5.2 (Theta correspondence). With notation as in Theorem ^. 1\ we have 
that 

E = {(F B (e(a)), Xe (a)) | a G P d } U {(F B (e(a)), x_ e(a) ) | a G P}. 

Moreover 

r(F B (e(a)),x e (a))) = £ 2 (~Pi + M a )) 

and, i/a£P, £/ien 

r(F fl (e(a)), x_ e(a) )) = £ 2 (-pi + 

Proof. By Proposition [EH (F B (e(a),x ±e(a) ) e E if and only if n ^ 0. 

Clearly 1 G W^_ d and — p x + p(a) = 1(— pi + p(a) + p c ) — p c is dominant for 
A + (v4 n _i), hence — pi + p(a) + p A is regular. This implies that (Fb(e(b), Xe(a)) G E 
for all a G P d . 

We now show that, if a G P, then (i^(e(a)), X_ e ( a ))) G E. First observe that, if 
P 7^ 0, then d = m < n. Consider s = S2s„_ d _ m+j - Then 

s(-pi +p(a) +p c ) = 

n— d— m+j — 1 ^ 

^ (n-m-i + -)5i + (-m - - + j)5 n ^ m+j 

i=l 

n—j 1 " 1 

+ 5^ (n-m-i + -)<J 4 + (n - m - z + - - a n _ i+ i)^. 

i=n— d— m+j'+l i=n—j+l 

Let ex be the permutation (in cycle notation) a=(n — jn—j — l ■ ■ ■ n — d — m + j), 
and let w be the element of VK(A n _i) such that w(Si) = S a ^y Then 

ws(—pi + p(a) + p c ) = 

n—d—m+j—l ^ ii— j ^ 

^ (n-m-i + -)5i+ (n - m - z - -)<!>;+ 

i=l i=n—d—m+j 



n 1 

^ (n-m-i-- - (a n _ i+1 - 1))^ = -pi + i/(a) + p c , 



i=n— j+l 



and, since a r > 1 for r > j, we see that ws(—pi + p(a) + p c ) is regular for A(A 



n— 1 . 



It follows that s G W^ d n 
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Suppose now that a G P<i\P. Then a r = for r > m — (n — d). In this case 

2(n-d) 

-pi + //(a) + p c = ^2 (n-m-i + -)5i 

n 1 

+ ^ (n - m - % + - - a n - i+1 )5i. 

i=2(n-d)+l 

If u> G W n -d, w 7^ 1, then write to = . . . S2<5 ifc with 1 < %\ < ■ ■ ■ < %\. < n — d. 
It is clear that 

0(-pi + 11(a) + p°), 5 h - S 2 ( n -d)-ii+l) = 

(-pi + p(a) + p c , -5 h - 5 2 (n-d)-n+i) = 0, 

showing that W^!? d = {1} in this case. This proves the first statement. 

For the second statement recall that, if a G P*, then —p\ + z/(a) = {s(— pi + 
fi(&)+p c )}-p c . Note that, if r(F B (e(a)),x± e (a)) = L 2 (\) then, by (jSJEj) , A(i7) > 
({w(-pi + p(a) + p *)} - p c )(#) for all w G W^_ d D W^l 9 d . Hence if we show that, 
for «, G n W^, w ^ 1, 

(-pi + p(a))(F) > (M-p! + p(a) + p^)} - p c )(H), 

and that, for a G P*, u; G W~_ d fl W^, u; ^ s 2 *„_ d _ m+ ., 

(-pi + i/(a))(tf) > (M-P! + p(a) + p c )} - p c )(H), 

we are done. 

Since v(H) = H if v G W(A n _i), it is enough to check that, for w G W n + _ d n>V^, 

(5.24) (- Pl + p(a) + p c )(#) > w(- Pl + p(a) + p c )(H), 
and that, for a G P*, w G fl W n % w ? s 2&n _ d _ m+p 

(5.25) s 25n _ d _ m+] (-p! + p(a) + p c )(H) > w{-p x + p(a) + p c )(H). 
If w = s 2 s n ■ ■ ■ s 2Slk , then (A, H - w(H)) = (A, -25 h 25 ik ), hence 

(-pi + p(a) + p c ){H - w{H)) = 2{n - d - i x + -) + ■ • ■ + 2(n - d - i k + -) > 0, 

so, using the fact that w = w' 1 , we obtain (15.241) . To obtain (15.251) we set jo = 
n—d—m+j and observe that, since a G P*, if jo < ik < n—d, then w(—pi+p(a)+p c ) 
is singular for A(A n _i). We can therefore assume that ik < Jo- 111 this case we need 
to evaluate (— pi + p(a) + p c )(s 2 s jQ (H) — w(H)), which is equal to 

(-pi + p(a) + p c , 2<5 io - 25^ 28 ih ). 

If i fc < jo then this is 2{n — d—i 1 +\)-\ \-2(n-d-i h +\) — 2(n-d- j + |) > 0. If 

4 = j then, since iu ^ s 2 s jo , we have k > 1 so (-p x + p(a) + p°)(s 2Sjo (if) - w(H)) 
is equal to 2{n — d — i\ + |) + • • • + 2(n — d — i^-i + |) > 0. The proof is now 
complete. 

□ 
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We conclude this section by observing that, as a byproduct, we have computed 
the character of r(rj) for 77 G £. In fact, combining Corollary 15.21 with Proposition 
EM we find (cf. (ICTj) for notation) 

(5.26) ch(L 2 (- Pl + /i(a)) = £ c„,cM/ 2 (M- Pl + //(a) + p c )} - p c ) 
and 

(5.27) ch(L 2 (- Pl + v( a ))= c w chV 2 ({w(- Pl +p( a )+p c )}-p c ), 

wew r n ^ d nw-_ d 

where c w = c w{ _ Pl+ll{a)+pC) . 

If A G (f) fl s^)*, let iy A be the subgroup of W(C n ) defined by Enright in [21 
Definition 2.1]. In our context, W\ is the subgroup, generated by reflections in 
roots a = 5i + 5j, such that: 

(i) (A,« v )gZ>°; 

(ii) /3g A, (A,/3) = (a,/3) = 0. 

Following [21 Definition 2.1], we introduce the root system A\ as the set of the roots 
a such that s a G W\. Set A A (A n _i) = A\ n A(A n _ 1 ), A+ = A+ n A A . Let also 
Wa(Ai_i) be the Weyl group of A A (A re _i). We denote by £\ the length function in 
W\ with respect to A^ and let be the set of minimal length coset representatives 
for W\(A n _i)\W\. In [21 Corollary 2.3] Enright proved a character formula that 
holds for any unitary highest weight module of a classical real reductive group. 
This formula is also implied by a stronger result proved in [3J. In the following we 
show how Enright's formula for L 2 (—pi + /x(a)) and L 2 (—pi + z/(a)) can be derived 
combinatorially from (15. 26p and (I5.27P respectively. 

Corollary 5.3. //a G F d and A = —p\ + p(&) + p c , then 

ch(L 2 (-p 1+ p( a )))= ^ (-iy^V 2 ({w(\ )} - p c ). 

Proof. Fix & e A and consider Ao, whose coordinates in the basis {5i} are of the 
form 

v = . . . , v n ) = (n - m + |, n - m - \, . . . , ~, -\ - bi, . . . , -\ - b m ), 

where bi = a m _ i+ i + i — 1. We say that a positive entry a in v is singular or regular 
according to whether —a appears in v or not. The group W\ acts on A as follows: 
it is the identity if the number of regular entries is less or equal than one; if there are 
exactly two regular entries, then the only nontrivial action is given by exchanging 
positions and changing signs of both regular entries; if there are more than two 
regular entries, it acts by an even number of sign changes of the regular entries 
followed by a permutation of the result. All other entries are fixed. 

If x G W^! 5 ^, then it acts on A by changing signs of some regular entries. Let 
w x be the permutation that arranges the result in decreasing order. Since the set 
Wn-d ^ Wli-d i s a se t °f cose t representatives for W\ (A n _i) in W\ , we see that 
W^ o = {w x x I x G Wn-d l~l ^n-d\ ( n °ti ce that this is true also when there are only 
two regular entries). Let w' x be the unique element of W(A n -i) such that w' x w x x(Xq) 
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is A + (A„_ ^-dominant. Since sgn(x) = 1, it follows from (I5.26P that 

ch(L 2 (-p 1 + /i(a))) = ^ sgn(w' x w x x)V 2 ({w x x(\ )} - p c ). 

To obtain our result we need only to prove that sgn(w' x ) = 1. We shall prove this 
claim by induction on n — m. Let s be the number of singular entries in v and 
let t > be maximal for which v n - m - t , . . . ,v n - m are singular. Set v' = w x x(v). 
We want to prove that arranging v' in decreasing order involves an even number of 
simple transpositions. If x does not change the sign to V\, then induction applies in 
a straightforward way. Otherwise we have v' = (v[, . . . , — v 1 , t> n _ m _ f , . . . , v n _ m , . . .). 
In making v' dominant for A n _i we should pass —V\ through the t singular values 
and the s negative values corresponding to negatives of the singular entries. So we 
obtain after s + t simple transpositions the element 

// / / \ 

v — 1,^1) • • • j ^n—m—ti ■ ■ ■ i Vn—my v«l^> ' 1> / ' 

s entries 

Consider now the vector 

u = (ux, . . . , u n ) = (n - m - \, n - m - |, . . . , ~, -\ - b[, . . . , -\ - b' m+l ) 

with hi — for i — 1, . . . , s, b' s+l = —x%, and b\ = for i = s + 2 . . . , m + 1. Let 
V = S2s 1 x and set u 1 = w y (u). Observe that v" is obtained from u 1 by performing 
exactly s — t simple transpositions. Hence v' differs from u' by an even number (2s) 
of simple transpositions, and we can apply induction. □ 

Corollary 5.4. J/aGP and \\ = —p\ + v(&) + p c , then 

ch{L\- Pl + i/(a))) = J2 (-l)^ (u,) ^ 2 ({^(Ai)} - P C ). 

A l 

Proof. Assume a e and let o be the permutation described in the proof of 
Corollary 15.21 Let w be the element of WA n _ t such that w(5i) = 8 a n) and set 
s = s 2 s n _ d _ m+j - Recall that ws(\ ) = X x . 

We now show that W\ 1 = wW^w' 1 . For this it is enough to check that a 
satisfies conditions (i) and (ii) for Ao if and only if w(a) satisfies both conditions for 
Ai. This is clear if (a, 5 n _d-m+j) = 0, so that s(a) = a. If a = 5 n _d-m+j + with 
n—d—m+j < i < n—j, then a does not satisfy condition (ii) and, if i > n—j then it 
does not satisfy condition (i). On the other hand (u>(a) v , ws(Xo)) = (s(a) y , Ao) < 
so w(a) does not satisfy condition (i). It remains to check the case when a = 
5i + 5 n -d-m+j with i < n — d — m + j . In this case one checks readily that a satisfies 
condition (i). On the other hand s{a) G A + (A n _i), so (s(a) v ,Ao)) > 0, hence 
(w(at), ws(\q)) > 0. If j3 is a root such that (j3,Xi) = 0, then (sw _1 (/3), A ) = 0, so 
(a, sw~ 1 ((3)) = 0. If (a, w~ l (f3)) ^ the only possibility is that w^ 1 (f3) = ±a, but 
this implies (s(a),X ) = 0. This contradiction implies (a,w~ 1 (f3)) = (w(a),/3) = 0. 
This shows that if a satisfies condition (ii) then also w(a) does. Reversing this 
argument we obtain that if w(a) satisfies condition (ii), then also a does. This 
proves our claim. 
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Notice now that w(A~l o ) = , thus = wW^w l . By applying f)5.27p we 
can write 

ch{L 2 (—pi + u(a)) = sgn(w x w x )chV 2 ({ww xs xsw~ 1 (\i)} — p c ). 

As shown in Corollary 15. 3\ sgn(w' x ) = 1, so we need only to show that sgn(w x ) = 
sgniwWxsXsw' 1 ) , or, equivalently, that sgn(w x ) = sgn(w xs ). Recall that, for y 6 
W re9 , w y is the unique element of Wx (A n -i) such that w y y(\o) is dominant for 
A^ o (A n _i). Since s(A ) is dominant for A^" o , it follows that w x = w xs and we are 
done. □ 

Remark 5.1. It should be noted that, even though the proofs of Proposition 15.11 
and its corollaries do not use classical invariant theory, they depend on Theorem 
14. 1| which uses in a crucial way the first fundamental theorem of classical invariant 
theory for 0(2m +1). 



6. THETA CORRESPONDENCE FOR THE PAIR (Sp(n) , SO* (2m)) 

In this section we use the denominator identity developed in the previous sections 
to derive the Theta correspondence for the compact dual pair (Sp(n), SO* (2m)). 
This dual pair, according to Proposition 14. 8[ corresponds to the distinguished sets 
of positive roots Aj 2 and Aj 2 , in a superalgebra of type D(m,n). 

We will develop the theory only for Aj 2 : the formulas corresponding to Aj 2 , 
are obtained by simply applying the reflection s em to the formulas corresponding to 

We have, for 1 < i ^ j < m, 1 < k ^ I < n, 

(6.1) A+ ={ei + ej, 5 k + 8 h 25 k | 1 < i ^ j < m, 1 < k ^ I < n}U 

{ej — 6j, 8k — 8i, | 1 < i < j < m, 1 < k < I < n}, 
A+ ={ei ± 8 k | 1 < i < m, 1 < k < n}. 

We have 2pi = (2n)(ei + . . . + e m ) and, as in type B(m,n), the defect of g is 
d = min (m,n). Set also 

A(D r ) = ±{e { ±e j \m-r + l<i^j< m}, 
A(C r ) = ±{5 k ±8 h 28 k \l<k^l< r}, 
A(A r _i) = ±{e k -ei\m-r + l<k^l< m}. 

Let W(A r _ 1 ),W(D r ),W(C r ) be the Weyl groups of A(A r ^), A(D r ), A(C r ), re- 
spectively. Set A + (C n ) = AfJ" n A(C n ), A + (D m ) = Aj n A(D m ), and A + (A m ^ 1 ) = 
Aq n A(A m _ 1 ). We denote by p c , p D , p A the corresponding p- vectors. Set 

(6.2) W r = subgroup of W B generated by {s €i s €j \ 1 < i < j < r}. 

With notation as in Section 13.11 the total order corresponding to this choice of 
A+ is 

ei > ■ ■ ■ > e m > 5i > ■ ■ ■ > 8 n . 
There is only one arc diagram associated to this total order, namely 

X = {e m 8i, . . . , e m -d+i8d}. 
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The corresponding maximal isotropic set of roots is S(X) = {71, . . . , 7 d }, where 
(6.3) 71 = e m - 5 U 72 = e m _i - 5 2 , • • • , Id = e m -d+i ~ $d- 

We want to apply Proposition 13.81 with B' = Supp(X). Note that, with notation 
as in lgXTl A(fi') = A(D d ) x A(C d ), and we have that A«(£') = A(C d ), so that 
W B /W fl (£') = VT(A d _ 1 )^(C (i ). With notation as in (Q, set 

Z = (^(A n _ 1 )/^(A d „ 1 ))W n _ (i (^(C , m )/^(C , d )) 

and, if m > n, 

^ = (VT(A-i)/W(A d _ 1 ))W m _ dSem _ dSem _ d+1 (^(C m )/^(C' d )). 

Then we can choose Z = Z if m < n and Z = Z U Zi otherwise. 
We define 

W = W(A m ^)W m - d W(C n ) 
so that we have Wq = W if m < n and while Wo — W U Wi with 

= (W(^ m _ 1 )/^(A d _ 1 ))W m _ dSem _ dSem _ d+1 W(^ d _i)^(C' n ) 

otherwise. 



If m < n, by Proposition 13.81 we have 
(6-4) e p R = fw r-r) ■ 



If m > n, Proposition 13.81 gives 



2ePR = (=3— — T-) + T Wx (; 



Lemma 6.1. If m > n then 

(6 - 5) fw ^..(ite-M)) = ^(l-e-M)' ' 

In particular (16 .4p /io/ds /or an?/ m, n. 

Proof. We need only to check that = .^^(Ppf)). Since right mul- 

tiplication by Wsf,4pp(x) stabilizes both and Wi, we can apply Corollary 13.61 
to both sides of (16.51) . Since odd reflections only change the sign of V(X), by 
applying a series of odd and interval reflection to both sides of (16. 5p . we are re- 
duced to checking that J r w(V(X')) = JF Wl (J>[X')), where X' is the arc diagram 

X' = {e m - n+i 5i\ i — 1, . . . n} whose underlying order on B is €\ > ■ ■ ■ > e m _ d+ i > 
^i > £ m ~d+2 > 82 > " " " > e m > 8 n . Let Y' be the arc diagram X' seen as a diagram 
on B" = {e m _ d+ j I 1 < i < d} U {5i | % — 1, . . . , d}. Since all simple roots of B" are 
isotropic, we have (px>, «) = (py, en) = for all a G A(B") so 

3F W (V(X)) = J r {w{A m ^ 1 )/w(A d _ 1 ))w m ^M Px '~ PY '^w(A d _ 1 )w(cA^( Y ')) 

Since all simple roots of B" U {e m _ d } are all isotropic, we see that (px', e m-d) — 
0. Clearly (py,e. m _ d ) = (7, e m _ d ) = for all 7 G Tnus ' setting s = 

Se .iS e , we have 

JWi(^W) = ^(w(A m _ 1 )/VK(A d _ 1 ))w m _ d (e Px ' _Py, ^sM/(A d _ 1 )iy(c' (i )('P(^ / )) ; 
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hence it is enough to check that FwiA^wic^iViY')) = 7* s w(^_ 1 )»'(c i )(^(i / '))- ^ 
is enough to verify the s £m _ d+1 -invariance. However, 

T w{Ad _ l)w{Cd) (V(Y')) = d\R Y/ e^' 

and this, up to a sign, does not depend on the choice of simple roots for A(Y'). 
Taking a system of simple roots containing 5i ± e, m _d +1 , we realize s em _ d as an 
automorphism of the corresponding Dynkin diagram so Ryie pY ' is s £m d+1 -invariant. 

□ 



For a G Prf, define 

fx(a) = - ^(a d+ i_ r )e m „ d+r , e(a) = ^ a r S r 



d 



r=l r=l 



If A G span(ei), set 



(6.6) chF A (X)= Yl s 9n(w)- 



e w(\+p A )-p A 



aSA+(A m _i) 



Arguing as in the previous section, it follows from (16. 4p that 

fan u/A+z^ chF A (w(-pi +/i(a) + p D ) - p D ) 

(6.7) chM (gi) = 2^ 2^ "Tf T\ _ ol\ chF c {e{&)), 

Fc(e(a)) being the finite-dimensional irreducible sp(n)-module with highest weight 
e(a). 

Recall from section H] that we constructed a real symplectic subspace V of Qi 
and a map $ : V — > (HP)* ®e HT m , such that there is a dual pair (Cri) G2) in 
sp(V, (• , •)) having the properties described in Proposition 14. 81 We look upon Sp(n) 
as a subgroup of Sp((lHP)* ®h EI m ) via its action on the first factor. Then G\ = 
{<3> _1 g$ I g G Sp(n)}. It follows that, if &~ 1 g& G Gi, then, since Spin) is compact, 
connected and simple, 

det w {$- l g§) = 1. 

Therefore 

Gi = Gi x {±1}. 

Given a representation (ir, V n ) of Gi = $ _1 S']?(r7,)$, we let (vr, Kr) be the repre- 
sentation of Gi on the same representation space such that 

(6.8) n(g 1 ,z){v) = z- 1 Tr(g 1 ){v). 

If Q is the set of finite-dimensional irreducible representations of G\, then we know 
from (14. 2p that the set S, occurring in Theorem 14.11 is a subset of Q = {tt \ n G Q}. 

Proposition 6.2. With notation as in Theorem \4-l\ we have that, if n G S, i/jen 
i/iere m a e such that 

tt = F c (e(a)). 

Furthermore, the \)-character of the isotypic component of Fc(b(3l)) in M A+ (gi) is 

(aa\ ST chF A(w(p D - fh± /i(a)) - p D ) 

(6-9) 2^ Tf (l-e-°) chFc ^ a ^- 

Proof. Clear from (16. 7p . □ 



54 GORELIK, KAC, MOSENEDER, PAPI 

In the case in question the set of roots of S2 is A(D m ). In the identification of f) 
with f)*, given by ( , ), we see that the element H G f) that corresponds to YmLi e * 
has the property that H^ v ± = ±1. Thus the parabolic subalgebra p 2 , defined by H, 
is 

aGA(A m „i) 

where n = ^2 (fJo)<* is the nilradical. 

aeA+(D m )\A(A m _i) 



We use the notation 15.11 Again we remark that 
chV 2 (X) = c x+p 



chF A (X) 



Corollary 6.3 (Theta correspondence). With notation as in Theorem ^. 1\ we have 
that 

E = {# c (e(a)) I a 6 P d } 

Moreover, if a G F d , then 

r(F c (e(a)))=L 2 (-p 1 + / i(a)). 

Proof. By Proposition 16.21 we need only to check that Fc(e(a)) G S for all a G P^- 
Clearly 1 G W m -d and — pi + p(a) = 1(— pi + /i(a) + p D ) — p D is dominant for 
A + (y4 m _i), hence — pi + p(a) + p A is regular. By Proposition 16. 2\ this implies that 
the isotypic component of Fc(e(a)) has a nonzero character, thus Fc(e(a)) occurs 
in £ as wished. 

For the second statement, arguing as in the proof of Corollary 15.21 we need only 
to show that, for w G W m _d with u>(— pi + p(a) + p D ) regular for A + (A m _ 1 ) and 
w 7^ 1, we have that 

(-P! + Ma))(tf) > (M-Pi + A*(a) + P D )} - P D )(#). 
Since w (if) = if if i> G W^A^-i), it is enough to check that, for w G W m _d, 

+ p(a) + p D )(#) > W (-p! + p(a) + p D )(H). 

This is clear because, if w = s eii . . . s tik , then (A, H — w~ l (H)) = (A, 2e^ + • • - + 2e ifc ), 
hence 

(-P! + p(a) + p D )(H - w~\H)) = 2(m-d-i 1 ) + --- + 2(m - d - i k ) > 0. 
The proof is now complete. □ 

As already observed in Section^, we have also computed the character of t(F c (e(&)). 
In fact, letting W r ^ 9 _ d be the set of w G W m -d such that w(—pi + p(a) + p D ) is 
regular for A(A m _i), then, combining Corollary 16.31 with Proposition 16.21 we find 

(6.10) ch(L 2 (- Pl + [!(*)) = c w sgn(w)chV 2 ({w(- Pl + p(8L) + p D )}- p D ), 



wew res >. 



where c. 



The argument given in Corollary 15.31 works also in the present case thus, with 
notation as in Section [5], we obtain 
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Corollary 6.4. If &eF d and A = —p\ + //(a) + p D , then 

c/ i (L 2 (-p 1 +Ma)))= (-1)^ H ^ 2 (MA )} - p D ). 



7. THETA CORRESPONDENCE FOR THE PAIR (0(2m), Sp(2n, K)) 

This is the most difficult case. Complications arise from the representation theory 
of even orthogonal groups, which we now discuss. Let V be the real symplectic 
subspace of $ji and $ : V — > M. 2m £g> (M 2n )* be the map such that there is a dual pair 
(G\, G2) in sp(V, (■ , •)) having the properties described in Proposition 14.81 We first 
parametrize the representations of G\. the same argument given in Section [5] implies 
that Gi is the group of pairs ($ _1 g$, z) with g G 0(2m) and z 2 = (det^2 m (g)) n . If 
Q is the set of finite-dimensional irreducible representations of G\, then, by (14.21) . 
we know that S C {-ft | tt G Q} where tt is as in (16.81) . 

To describe Q we need to recall that G\ ~ 0(2m) is isomorphic to the semidirect 
product of Z/2Z with SO (2m). The generator x of Z/2Z in Gi can be chosen so 
that Ad(x) induces on h fl the reflection s tm . The finite-dimensional irreducible 
representations of Gi are determined by their restriction to the connected compo- 
nent of the identity G\ ~ SO(2m) of G\ and by the action of x. First of all, we 
parametrize the irreducible finite-dimensional representations of G\ as irreducible 
finite-dimensional representations of s^: having chosen f) Dsf as a Cartan subalge- 
bra, we choose Aq H A(s^) as a set of positive roots for sf, where Aq is as in (16. ip . 
and let p D be the corresponding p-vector. Given A G (h fl )* dominant integral, 
we let .Fd(A) be the irreducible finite-dimensional s^-module with highest weight A. 

If A = J^r=i a i e i w ith di G Z and a\ > a<i > ■ ■ ■ > a rn > 0, then there is a unique 
irreducible representation of G\ that restricted to G\ contains -Fd(A). We denote 
this representation by F + (X); it can be checked that the restriction of F + (X) to G® 
is 

F+(\) = F D (\)®F D (s em (\)). 

If instead a m = 0, then there are two irreducible representations of Gi whose 
restriction to contains Fd(X). Both restrict to G\ as Fd(A), and we let F + (X) 
be the one such that x acts trivially on the highest weight vector, and F~(X) the 
one such that x acts by det^2 m (x) = — 1. Note that this parametrization depends 
on the choice of x. 

We also need Kostant's generalization to nonconnected groups of the Weyl char- 
acter formula, given in Theorem 7.5]. In the particular case at hand, we let T be 
the torus in G\ such that the Lie algebra of T is contained in [). Set H + = T U xT. 
The group H + is the normalizer of the torus T and of the Borel subgroup of G\ 
corresponding to our choice of positive roots. Then Kostant's formula in this case 
reads 

(7.1) ch F+{x) (x s t) = 1 + { ~ ir chF+(\)(t) 
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if A = Y^iLi a i e i with a m > 0, while, if a m = 0, 
(7.2) ch FHx) (x s t) = 1 + { ~ iy chF D (\)(t) 

where W / (C m _i) is the Weyl group of the root system 

A(C m _i) = {±(e* - ej) | 1 < z < j < m - 1} 
U {±(ej + ej) | 1 < z < j < m - 1}. 

Note that, since p D = Yll=i ( m ~ e «> 

^ sonHe^ = (1 - e~ Q ), 



where 



weW(C m -i) aeA+(C m _i) 

A + (C m _i) = {(ei - €j) | 1 < z < j < m - 1} 
U {(ei + ej) | 1 < z < j < m - 1}. 



Set f)2 = f) H Sj. We need to compute the if + x f) 2 -character of M A+ (gi). In this 



case 



= {5 k ± €i | 1 < z < m, 1 < fc < n}, 
and 2pi = (2m) (o\ + . . . + 5 n ). 

Let ad be the adjoint action of g on fli- Recall from Section 0] that, as G\- 
module, M A+ (fli) is given, up to the action of the center, by the action of G\ on a 
polynomial algebra P(W). The differential of this action on W coincides with ad* s c 

on (flj")*. Hence we can identify P(W) with the symmetric algebra S(Qi) with 
acting by ad. Let us denote by Ad the action of G\ on S($f[) coming from its action 
on P(W). We normalize the choice of x by assuming that we can choose the root 
vectors X-Si±ej i n Qi i n suc h a wa Y that 

Ad(x)(X- Si±e .) =X- Si±ej 

if j < m, and 

Ad(x)(X_ 6i+e J = X_ Si _ em . 

Clearly, if t G T x h 2 , then 

c/^+ xf , 2 M A+ ( 01 )(t) = = !^__(t). 

In order to compute ca^+ X f )2 M A+ (g 1 )(xt) we simply compute the trace of the 
matrix of the action of xt in the basis given by monomials 

If A is the f)-weight of this monomial, then 
Ad{xi)([[X^ i+e X%_ ej ) = e\t)( X a _l +e X%_ ej X^ +e JC^_J. 
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Thus the only contribution to the trace is given by the monomials 

J 

It follows that 



l<i<n l<i<n 
l<j<m 



1 < j < in — — 

Putting everything together we find that 

(7.3) ch H+xt)2 M A+ ( Q ^)(xH) = 1 + -77— —^tW 

1 - (-l) s e~ pl 
+ 2 ffe (l-e-^)ni<i<„(l-e- M (t) " 

1 < j < m — — 

We now apply our denominator formulas to the two summands above. We start 
with the first one: again the defect of Q is d = min(n, m). We set 

A(D r ) = ±{e, ± e,, | 1 < % ± j < r}, 
A(C r ) = ±{5 k ±5i,2S k \n-r + l<k^l <n}, 
A(A r _i) = ±{5 k -5i\n-r + l<k^l<n}. 

Let W(A r _i), W(D r ), W(C r ) be the corresponding Weyl groups. In this case 
2 Pl = 2mJ2ti^- Set A+ ( C n) = Ao H A(C n ), A+(B m ) = A+ n A(JB m ), and 
A + (74 n ,_i) = Aq n A(A n _i). We denote by p c , p B , p A the corresponding p-vectors. 

With notation as in Section 13.11 the total order corresponding to this choice of 
A+ is 

8 X > ■■■> 5 n > €i> ■■■> € m . 
There is only one arc diagram associated to this total order, namely 

X = {6 n e lt ... 

The corresponding maximal isotropic set of roots is S = {71, . . . , 7^}, where 

(7.4) 71 = 5 n - ei, 72 = 5 n _i - e 2 , . . . , 7d = 5 n ^ d+1 - e d . 
Applying formula (I1.15P we can write 

e p+£iiM 

(7.5) eP R = ^ ). 

LUil 1 -e V 

Recall from (15. 8p that W r is the subgroup of W 5 generated by {s2s, \ i — 1, . . . ,r}. 
Define 

Wy Xt = reflection group generated by W g and the reflections s ev 
W(B m ) = subgroup of W^ xt generated by W(D m ) and s ev 

Extend sgn' to W^ xt by setting sgn'(s ei ) = 1. We define 

W = W{A n . x )W n . d W{B m ), W D = W(A n ^)W n - d W(D m ) 
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If n < m then = W(D m ) hence, summing over W(A n _i), we find 

(7.6) d\e p R = J t w J— j ). 

Since s e fixes we can rewrite 07.61) as 

n i =i( i - e 7i ) 

e p+EiiM e p+EiiM 

dlePR = ^TWTi ^ - ^TWTi ■ 

rii=i(l - e V lli=i(l -e *) 

The same argument given to prove (I5.12p gives 

(7.7) e p R = J= w {—. )-jt w ( ). 

If n > m then = W{C n ). By an explicit computation we see that p+X^Li 
Set 

(7.8) 3^ = subgroup of W* xt generated by {s 2 s n _ i+1 s ei | 1 < % < d}. 

We have that s 2Sjl _ %+1 s e% (p + E?=iM) = P + E?=iM hence 

e p+EiiM eP+Etihd e p+Eti]Til 

sV(s 2 5 n _ i+1 S e Js 2(5n _ 



1+1 €i n d u 1 -^) nt=i(i-^) n^(i-^: 



It follows that, if w = nlU s 25„_ d+lr s e , r G 3^, then 

e P+EtihiI _ IT1 pP+Etihd 

Note that if | J\ > 2 then there is a reflection in W(A m _i) that fixes the element 
hence 



e P+E? = il7i 



e 



p+EtiM . P p+EtJ74 * pp+EtJ^D 



J r w (sgn'(w)w — -j ) = JV( — 3 / ^ff t; rj- 



Note that 



e P+E-=iM e P+E-=iM 



hence, if w G 3^d, we have 



e p+EiiM 
Jmt sjn'Hui- j— — -) 

e p+EtiM e P+EiiM 
= J^wi — i k — -j— ; ). 

mi(i-^o nK(i-^r 
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Using the fact that W£ xt = Wy d and that s ti e p R = e p R, it follows that 

2 d d\e p R = 

e p+Etihd e p+EiiM 



2 d ^Vi^( — ^ ) — 2 rf 1 dJ 7 w( — -j— ; ). 



Hence 

e p+Etihd 1 e p+Etj74 
(7.9) ci!e^ = ^(- r - ) - ^(-3=!- — ). 

Set W = W^.j+ijed]- Using the fact that W n -<i and W commute, we can choose 

W/W = ((PU(A n -i) x W(B m ))/W')W n . d . 

Let Y be the arc subdiagram corresponding to the interval [S n -d+i,€d]- Applying 
(I1.15p . we have 

e P+EiiM e py+Etil7iI 

^(-r- — ) =?wiw>{e p *- pY ?w\—r- -)) 

ll^il 1 -e 7j ) ll^il 1 -e 7 

= J e 'iy/w'(e px " py rf!e py J Ry) 

Observe that ^(7) = 2% - 1 so nil ht{ ^ +1 = d\. Applying (EU) to Y we find 
•M— 3 ) =^w(e px ~^cflJV'(^ r-r-))- 

v n? =1 (i-^) ; 1 mi(i-e-Mr 



hence 

pP+EiiM „ e P 

Analogously one checks that 

e p+Ei=il7iI v e p+hdl 

(7.11) Fwi— n ) = ; n r-r)- 

Observe in (17.1 ip that ]7d[= [7^-1], hence 

e P+lldl e P gP+l7d-ll 

(7.12) r w (—— r~r) = ^w(—n r~r) +^w(—r^ r~r) 

and the second summand in (17.121) is zero since the reflection s £d _ 1 _ ed fixes it. Plug- 
ging these formulas in (17. 9p we find 

(7.13) e p i? = Twi— 3 r-H ~ »^W( 5 i r-r)- 

Finally, observe that W = Wd U H / z)S em and s em fixes ..^ eP _ [7 .] , so 

(7.14) e p R = f w (— -. — ) - TwA— tt— — ). 
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To give a uniform treatment of the cases n < m and n > m, we set di = min(n, m — 
1) so that f )7.14p and ( 17. 7p combine to give 

(7.15) e PR = f w (— )-jt ( e l y 

Set P+ = {a G F d \ a m > 0} and P° = F d \Fj. Clearly, if d < m then P+ = and 
F d = Fd- Dividing (I7.15P by V = e p0 Yl a eA + (^ ~ e ~ a )> we find that, arguing as for 
type B(m, n), 



Since £> = C c ELeA+ta^ 1 ~ e °) eP " n a6 A+(D m )( 1 ~ e % we can rewrite the 
above formula as 

p-pi 

(7.16) 



rL eA +(i-e- Q ) 

E E ^H C "n (W( " P1 + Ma) + <P 

aeV +weW n _ d lla6A+(0,)\A+(A,_i)V A e J 



, chF A (w(-p 1 +h(sl) + p c ) - p c ) 
+ 2^ 1^ sgn ( w >~- rf n - -a\ chF D{e{*))- 

a& o weWn _ d llaeA+(C n )\A+(A„_ 1 )l i e ) 

Here we recall that, if a e P d , then //(a) = - ^ =1 (a rf+ i_ r )<5 n _ d+r , e(a) = Y^t=i a r £ r, 
and c}iFa(\) is given by (15.161) . 

We now take care of the second summand in (I7.3p . Note that 

l<j<m — — 

rii<*<- (1 -e"^) 

1 < j < m 

and that 

n m—1 

P C + P D ~ Pi = E( n - ( m - !) - + E^ m ~ 

i=l i=l 

so the formula above is precisely the denominator for the distinguished Borel sub- 
algebra of type A^ 2 for a superalgebra of type D(n,m — 1). 
We can therefore apply the results of Section [6] and find that 



n il <^(i-e-^)n 1<! <n(i-^ 
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where W(m - 1) = VT(A n _ 1 )W n _ (il ^(C m _ 1 ). Recall from f lo~TUj) . that W r + 

(resp. W r ~) is the set of u> G W r such that w = s 2 s i ■ ■ ■ 82s, with k even (resp. 
odd). 

By dividing off e pC ]laeA+(C7„)( 1 ~ c~ a )e pD riaeA+(c m _ 1 ( 1 ~ e ~°) and expanding 
the denominator, we find 

(7.17) ch H+xt)2 (M A+ (Q 1 ))(xt) = 

/ ^ chF A {w{-p 1 + fi(&) + p c ) - p c ) 

E wet y ( c m - l} ^(^)e w(£(a)+pD) 

x E^'W^ Uj ' 

Recall that, if n is a finite-dimensional representation of Gi, then 7r is a repre- 
sentation of Gi whose definition is given in (16. 8p . We can now state 



Proposition 7.1. With notation as in Theorem 4A, we have that, j/jf G S, then 
there is a £ such that 

7r = F ± (e(a)). 

Furthermore, if a G Pj", i/ien £/ie x ty 2 - character of the isotypic component of 
F+(e(a)) m M A+ ( Sl ) 

/ n chF A (w(p c — p\ + «(a)) — p c ) , . .. 

(7.18) s 9< w ) n /_ _;/ cfe g+ F + (e(a)), 

*W„_, llaeA+(C„)\A+(A n _ 1 )l i e J 

while, for a G P°, £/ie if + x ^-character of the isotypic component of F ± (e(a)) is 

ch F A (w(p c - pi + M a )) - P C ) 

LeA+(C„)\A+(A n _i) 



/ 71 m , ,chF A (w(p c - pi +A*(a)) ± 

(7.19) 2^ s S n Mn n _ -a\ ° h H +F ( £ ( a ))> 

...^v»± ll Q eA+(C„)\A+(A n _i)U e J 



Proof. It follows from (I7.16P that i^Q^Li a « e «) can occur in M A+ (gi) only if a, = 
for i > d. This proves the first statement. 

It is also clear from (I7.16P and Kostant's formula (17.11) that, if a G Pj, then the 
isotypic component of F + (e(a)) is given by (I7.18p . 

If a G P°, let M a be the sum of the isotypic components of F + (e(a)) and F~(e(&)). 
By substituting f[7Tl6]) and (ITTTj) in (Q, we find that 

ch H+xt , 2 (M a )(x s t) 



l + (-l) s , ,c/iF^( W ( 
2^ sgn(w)—. = 

Z „ lC vu , llaeA+ 



chF A {w(-pi + //(a) + p c ) - p c ) 



«,ew n _ d llaeA+caoyv+^-oC 1 e " ; 

„, cw + lLeA+(c„) 

n — di 



x cM^D(e(a))(t) 



Observe now that, since a G P°, then s_25 n _ dl fixes — p\ + /u(a) + p c so, if 
w ' s -28 n _ dl with «/ G W~_ d then sgn(w) = —sgn(w') and w(—pi + p(a) + p 



w 
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vu'(—pi + /i(a) + p c ). Combining this observation with Kostant's character formula 
(j7.2p . we can rewrite the formula above as 

ch H+xt)2 (M a )(x s t) 

E, .chFA(w(—pi + u(a.) + p c ) — p c ) , , . . , u/ , , 



n ae A+(ao\A+(A„_ 1 )^ 1 e 

n — d 

E, . chFA(w(— pi + u(a) + p c ) — p c ) , 
s^n(ti;) M 1 - ^ J M P ' J! (F- (e(a)) (x*t) 

...evw- llQeA+(C„)\A+(A n _ 1 )l i e J 



from which (I7.19P follows readily. □ 

In our case the set of roots of S2 is A(C„). The element H e f) that corresponds 
to — X)ILi has the property that H\ v ± = ±1. Thus the parabolic subalgebra p2 
defined by H is 

P2 = f> © 5^ (flo)a © n, 

aeA(A n _i) 

where n = (flo)a is the nilradical. 

aeA+(C„)\A(A„_i) 

Recall that, if P is as in (I5.22p and a G P, then we can define the weight z/(a) as 
in (I5.23|) . Using Notation 15. 1\ we can now state 

Corollary 7.2 (Theta correspondence). With notation as in Theorem \4-l\ we have 
E = {F+(e(a)) | a e P d } U {F"(e(a)) | a e P}. 

Moreover 

r(F + (5(a))) = L 2 (-p 1 + / ,(a)), 

an<i, ija£ P, i/ien 

r(r( £ (a))) = L 2 (- ft + Ka)). 
Proof. Using Proposition 17.11 the proof follows exactly as in the proof of Corollary 

E2J □ 

As already observed in previous sections, we have also computed the character 
of r(F ± (e(a)). We set, as before, W^!? d to be the set of w G W„_d such that 
w(-p! + /i(a) +p c ) is regular for A(A n _i) and, for w G W^ d , c w = c w( _ pi+At(a)+p C). 
Then, combining Corollary 17.21 with Proposition 17.11 we find that, if a G Pj, then 

(7.20) ch(L 2 (- Pl + /i(a)) = ^ c„,s^n(w;)c/iV r2 ({u;(-pi + //(a) + p c )} - p c ), 
while, if a G Pj~, then 

(7.21) ch(L 2 (- Pl + //(a)) = ^ c w c/ i y 2 (M-p 1 + /i(a) + p c )}-p c '), 

wevv^L^nvCfd 

and, if a G P, then 

(7.22) ch{L 2 (- Pl + z/(a)) = - £ ^cW 2 ({«;(-/>! + //(a) + p c )} - p c ). 

^ew-_ d nw;i s d 



The argument given in Corollary 15.31 works also in the present case, thus, we 
obtain 
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Corollary 7.3. //aePj and A = —p\ + /i(a) + p c , then 

cML 2 (-p 1+/ i(a)))= £ (-lY x ° iw) V 2 ({w(\o)}-p°). 

I/aGP and Ai = — p\ + z/(a) + p c , i/ien 

cML 2 (- Pl + Ka)))= J2 (-1) %H ^ 2 ({^(Ai)}-P C ). 

a i 

8. THETA CORRESPONDENCE FOR THE PAIR (U (n) , U (p , q)) 

In this section we use the combinatorial machinery developed in the previous sec- 
tions to derive the Theta correspondence for the compact dual pair (U(n), U(p, q)). 
This dual pair, according to Proposition 14.81 corresponds to the distinguished sets 
of positive roots A^' 9 ^ in a superalgebra of type gl(m, n), p + q = m. 

Let g = gl(m, n). Its Weyl group is 

W s = group generated by s ei _ e2 , . . . , s em _^ em , s Sl _ S2 , . . . , ss n _^ Snl 
which we identify with S m x S n . Recall that the defect of q is d = min(n, m). Set 

A(A m _i) = {ei - €j | 1 < i ^ j < m}, 
A(A„„ 1 ) = {5 k - S t | 1 < k ^ I < n}, 
A(A p _i x = {e k -ei\l<k^l<porp+l<k^l< m}, 

A c = A(Vi x Aq-x) U A(A n _i). 

Let W(A n -x) be the Weyl group of A(A n _i) and ^(A^i) the Weyl group of 
A(A m _i). Let W c be the Weyl group of A c (the subscript "c" stands for compact). 
Set A+(4»-i) = A+ n A(A»-i), A+(A m _x) = A+ n A(A m _i), and A+(Vi x 
= Aq fl A(A p _x x We denote by p^™- 1 , p" 4 ™- 1 , pP A the corresponding 

p- vectors. 

Set 

L = {ie {0,...,p}|0 < d-i < q}. 
Set i max = min(n,p) and j min = d - i max . For cr e S m set 

m(a) = \{a(i) \p- imax + 1 <i < p + j m in} n {1, . . . ,p}\, 
Ui = {w G S m \ ni(a) = i}. 

Then S m = LL eL £4 

With notation as in Section [31 the order corresponding to A^'^ is 

e t > ■■■ > e p > 5i> ■■■ > 5 n > e p+1 > ■■■> e m . 

Define, for i G L, f}\ = e p ^ t +i — 5 t foi 1 < t < i and Y t = 6 n -t+i — € t+p for 
1 < t < d — i. For i £ L, let be the arc diagram 

Xf = {^_ m <5 t | l<t<i}U{5„_ 

H-i e p+t I l<t< j}- 

Then 

(8.1) S(X;) = {# I 1 < t < i} U {f t | 1 < t < d - i). 
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We start by writing formula (I1.15P corresponding to S(X imax ), which, for sim- 
plicity, we denote from now on by S_. Let W{ (resp. W£) be the subgroup of S m 
that fixes 1, . . . ,p — i and p + d — i I + 1, . . . , m (resp. p — i + 1, . . . ,p + d — i). Set 
Wi = W{ m ™ and W 2 = W^. Then Ui is stable under the right action of W x , W 2 , 
and under the left action of S p x S q . Choose VjCf/j such that Ui = V^W\. Let W[ 
be the subgroup of W\ stabilizing Vi and set V- = VijW[. 

Assume that n < m. By the above, using notation of Section II. 2[ 

Re p = F Sm (Q{X imax )) = X)^(G(^—)) = E^'(^(Q(X W J). 

ieL ieL 

Set v = e D _i + i, w — e D _i +d\ let Y be the subdiagram in X, which corre- 
sponds to the interval [v, w]; note that S(Y) = S(X imax ) and each vertex of Y is an 
end of some arc. Moreover, W[ ViW ] = S n x W\ and we may choose W* for Y to be 
either or W±. One has 

^(Q(^wJ) = e"-^JF Wl (Q{Y)) = eO-»-f Sn (Q(Y)), 

hence 

= TyuT\ (p^Vi {-Fsn (Q G^imo* ) ) ) ■ 

Since the elements of Vi C £ m commute with the elements of S n we conclude that 
(8-2) Re" = ^(^(^(Q(X W J)). 

Note that, if m < n, then L = {p}, U p = S m = W\. Thus (18. 2 p holds as well with 
V p any subset of S m , being just (I1.15P for this case. 

We now make a careful choice of Vi by exploiting the action of W\ x W 2 on Ui. 
Set cTj = TT+^T^Se ,,- e Note that cr* is an element of tA. Let O be 

the orbit of (5* p x S q )ai under the right action of W 2 in (S p x S q )\Ui. First we 
observe that, if x, y are distinct elements of O, then xWi fl yW\ = 0. Indeed, if for 
a G PV2 we have (S p x S q )<Ji(T ^ (S p x S q )<Ti, then (5 P x S^a^a 7^ (5^ x S^Oi for 
any 77 G Wi. This can be checked as follows: assume OiOOi S p x S q , hence there 
is i G {1, . . . ,p} such that <Ti<T<Ti(i ) > p. If i < p — i max then, since ai(i ) = i , 
we have a(i ) > p + d — i, so airjaa^io) > p for any 77 G Wi. If p — i < i$ < p then 
criCrcrj(io) = io so we can assume i = p — z max + i with t < i — i max - In this case 
°i (^0) = P + Jmm + * and we must have that a(p + j m i n +t) > p + d — i for otherwise 
we would have GiGaiiio) < p. But in this case rjaa^io) > p + d — ifor77G Wi and 
we are done. 

Next we show that the action of Wi x W 2 on (S p x S q )\Ui by right multiplication 
is transitive. For this it is enough to show that for any w G S m C\Ui, there is a G W± 
and r G VT 2 , 77 G (S^ x S q ) such that (Tj = rjwar. Since |{w(t) | p — i max < t < 
P + jmin} n{l, . . . ,p}\ = i, we can find a G Wi such that wa(t) < pior p — i < t < p 
and wa(t) > p for p - i maa . < t < p - i max + i or p<t<p + j min . We can 
find 77 G (S'p x S'q) so that rjwa(t) = t for p — i < t < p + j m i n . Moreover we can 
also assume that r]wa(p — i max +t) = p + j m i n + t for 1 < t < i max — i- Finally, 
we can find r G W 2 such that r)war(t) = r)wa(t) for p — i max < t < p + j m i n , 
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r]war(p + j min + t) = p - i max + t for p + j min < t < p + d - i and r]war(t) = t 
otherwise. This means that rjwar = as wished. 

We set y i m _ d to be a set of coset representatives for Stabi\W 2 where Stabi is the 
stabilizer of the coset (S p x S g )<Ti under the right action of Wi on (S p x S q )\Ui. 

From our discussion it follows that we can choose K = (S p x S q )aiy i m _ d . Since 
y^n-d c ^2, w e see that is the subgroup of Wi stabilizing (S p x S q )a i) thus 
OiWl<Ji is the subgroup of W] stabilizing 5^ x S^. It follows that aiW^ai = W{ fl 
(S'p x S'g) ~ SiX Sa-i- In particular \W(\ = i\(d — 

We can rewrite (18.21) as 

Set = ^yL-d^i- Since ^m-d c W 2, it is clear that W m _ d C W|. We can 

therefore rewrite (18.31) as 



(8-4) e'R = Y,^w c w }n _ d {sgn(a i )a l Q(X irn J), 

Lemma 8.1. Given i G L, set Ti = i max — i. Then 

(8.5) ^n(^(Q(X imo J) = ^(PU-I^DgpQ 

Proof. By an explicit computation, we see that 



P~ 1 o P 



m — n — 2r + 1 q — p + i— j — 1 



E v-^ lib — lb — en T x ^— \ 

hi = 22 2 er+ E " " 2 " — £r 

7eS(Xi) r=\ r =p-i+l 

2 Cr+ E 2 e ' 

r=p+l r=p+j+l 



n 



+ 2_, 2 ^ + 2^ 2 

r=l r=i-\~l 

sr(Jfi)l7[> hence we have 

e P+E 7 eS(A- i )l7[ 
e P+E 7eS (A- i )h[ 

e P+E 7eS (x i _ 1 )M+]/9!H7ti+iI+^+<i-i+i-^ 

FJ^(1 — e -ep-r+l+<5r) [T d I* +1 (l — e ep+r-5n-r + l) 

Since ]#[= [Ci 1 '] and h d Z] +1 l+e P +d-i + i - = Uz] +1 l we obtain that 

e P+E 7eS( x,)l7[ e P+E 76S( x 1 _ 1 )h[ 

8.6 s e _— - = e lPi-il-h d -i+il 



n (i-^) n a-e-T) 



Since 



1 ([/3-]-K_J) = [/3 4 -i 1 ]-K:U 



the lemma is proven by an obvious induction. □ 
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Combining (18. 4p and ( 18. 5p . we get 

One last computation is needed: 
Lemma 8.2. 

1 r r ■ « pP+n(I/9?l-[7Lj) 

< 8 - 8 > ^(^^..(e-w-M-J'at^.)) = W ^ ( n — ^ _ e _ M) ) . 

Proof. Set the subdiagram of Xj whose support is [e p _i+i, 5j] and Fj 2 the subdia- 
gram having support [5 n -d+i+i , ■ Let d = W [ep _ i+lA ] and G 2 = W^^!,^,,^ 

An easy calculation shows that 



p+d—i 

h=p— i+1 h=l h=n—d+i+l 



In particular — [7^_J is Gi x G 2 -invariant. Since Gi x G 2 C W c and commutes 



with W^_ d , we can write 



Combining ([32]) with ( ITTSj) . we have that, for any g, JF W& (Q{X)) = P Wb {V(X)), 
thus 

= ^c^(We r<(W, - W - J) ). 
The lemma follows from the observation that V{Xj) = i\(d— z)! j=j e ^ 1 _ e _ [7] ^ . □ 

Using fl8.8p . we see that ( 18. 7p becomes 

Using f !8.9p . and expanding in geometric series in the domain f) + = {h G f) | 
a(h) >0,a6 ^t}, formula ( 18. 9 j) becomes 

(8.10) e'* = £^_ d ( E e^-^- 4 )), 

where 

V(a, b) =p - E a i-*+i e p-H-« + E ^ e p+* + E ~ E b d-i-t+i8 n -d+i+t 
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and we denote by r k the partition (r, r, . . . , r) G P*.. Note that ( 18.1 Op can be written 



as 



(8.11) e'R = J2F w , w . m J Y, ^"^y 

i&L aePi.ri^-'CbePd-i 

From the formula above we can already derive an explicit form of the Theta 
correspondence, but, in order to have the same parametrization of [32], some extra 
work is needed. For a G Pi and k < % define 

f k (a) = (ai, . . . ,a fc ), 

l k (a) = (ai-fc+i, . . . , a*). 

If a C r l , we set r*\a = (r — a,j, . . . , r — ai). If a G Pj then, given 6 G P^_j with 
af~ l C b, we let b U a = . . . , bd-i, ai, . . . , fli). 

Given r G N and a G Pj, let fc G {0, ... , z} be such that r k C a. If (r — ai) d_ * C b, 
then we set 

s fc (a, b) = (/ fc (a) - r k , h U (r^V^a))). 

We let W, l n _ (i (a, b) be the set of v G W^_ d such that u(V(a, b)) is regular for 
A(A p _! x Given r G N and a 6 Pj, let k r (a) be maximal in the set of 

k G {0, ...,«} such that r fc C a. 

Lemma 8.3. Assume that m < n and fix i G L. Lei (a, b) G Pj x P^_i be such that 
(r, — ai) ' -1 C b and set k = k r .(a). 

(1) If k G" L, then v(V(a. — r/, b)) A c -singular for any v G W* n _ d . 

(2) If k £ L then there is a bijection v \-t z between W^__ d (a — ?Y,b) and 
yV k _ d (s k (&, b)) stzc/i t/iat 

J- w >^K y(a - rAb) ) = JV c (5yn(z)e* v ^^ b »). 

Proof. We will prove both statements by induction on % — k. The case % = k is 
obvious. 

If < i, set to = P — i + 1 — iji — a>i)- Note that p — i max + 1 < t < p — i. 
If p + d — i — m then, since k < i, k (jL L. Moreover we have that we can choose 
Wm-d = {!}■ An eas y calculation shows that 

p i i m i i 1 n iii 

E.m-n + 1 v^.n + m + l ^r-^.g-p + n + 1 

( g )Q+ S ( 2 *) e ' + ZJ^^ 

t=i t=p+i t=i 

so (V(a, b), e to — e p _i+i) = 0. It follows that t> (V(a — r^, b)) is A c -singular for any 

v eW i 

If p + d — % < m, let s G 5 m be defined by setting s(t ) = p + d — i + 1, 
s(p+d — i + 1) = p—i+1, s(p — i + l) = t , and s(t) = t for t 7^ to,p— i+l,p+d— 
Then s(V(a,b)) = V{a! - (n + l) i_1 ,b') where a' = L„_i(a) + V~ l and b' = 
b U (r, — Oj) 1 . Suppose that v G Y\? l m _ d and assume that i;(V(a — r/, b)) is Ac- 
regular. Then f (to) > p + d — i, so vs _1 (p — d + z + 1) = s > p + d — i. Hence 
«>' = s ep _ d+i+1 - eso ^ -1 e W^r 1 and 

T Wc (sgn{v)e vV ^^) = ^(asnMe^'-^'W). 

This implies that there is a unique w G W^ d (a' — (rj + b') such that 

F w Xsgn{vy v ^V) = J Wc (sgn(w)e wV(a '- (r ^- 1 ^). 
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Clearly the argument above can be reversed to prove that the map v H- w is 
a bijection. Since r j + 1 = rj_i, (rj_i — aj_ 1 ) d_l+1 C (rj — aj) d ~ i+1 C b', and 
fc rj _ 1 (a / ) = fc ri ( a ), w e see that we can apply the induction hypothesis. Thus, since 
w(V(a' — (rj + b')) is A c -regular, we have k £ L. Moreover there is a bijection 
iw ^ 2 from W m _/ _1 (a' - fa + 1) <_1 , b') to W*^(s fc (a', b')) such that 

F Wc (sgn{w)e^'-^ i - 1 ^) = T Wc (sgn(z)e zV ^ a '^). 

Since it is clear that Sfc(a',b') = sj.(a, b) we are done by composing the two bijec- 
tions. □ 

Proposition 8.4. 

e p R = F Wc wima* ( ^ e v{a ' h) ) + ^2 ^2 e V{a,h) )- 

aeW imax ,heF jmin i£L,i<i max aeP i ,bePS_ i 

Proof. The expression (18.111) for e p R can be written as 

FwcW^i e F(a,b) ) + ^2 ^WcW^i ^2 e y(a,b) ) 

+ ^WcWl^ ( e y(a_ri ,b) ) . 

Let /// denote the third summand. By Lemma 18.31 we have 

/// = ££ E ^ , *" Kb "). 

i£L k£L,k<t r^CaePnr^+Va 

bePj r.^'Cb 

which is 

k£L,k<i max i&L,i>k r i k Ca£F i ,r i k+1 <£a 

We observe that the map is a bijection between the set 

|J {(a, b) £ Ft x P d _ 4 | k n (a) = k, rt l C b} 

and the set 

{(a ) b)eP fe xP*|(w-fc)^b} J 
hence we can rewrite /// as 

V T k ( V P V((a,b))\ 

keL,k<i ma x a£Pfc 

Upon substituting we get the desired statement. □ 
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Arguing as in the previous sections, it follows from Proposition 18.41 that 
chM A+ ( 9l ) = 

ch{F M (v(-p! + //(a, b) + p^- 1 ) - p A ^)) 



nQ6A+( J 4 m _ 1 )\A+(yl p _ix J 4 9 _ 1 )( 1 e 

s9n ^ 

ieL,i<i max aSP,,beP* veWf n _ d 



-chF n (e(&, b)) 



cft^M-P. + Ma,b) + - P A "-)) cft fn(E(a b) 



na e A+( J 4 m _. 1 )\A+(A p _ 1 x,4 g _ 1 )( 1 e °) 

V 

/i(a, b) = — aj„ t+1 e p _j + ( + b t e p+t 



Notation is as follows: A+ = A^' 9) ; if a £ P«, b £ P d _ i; we set 



d—i 



t=l t=l 

rf- — -i 



t=l t=l 

Here F n (e(a, b)) is the finite-dimensional irreducible w(n)-module with highest 
weight e(a, b). Finally, if A £ span(ei), set 

e w(\+pP>*)-pP'i 

chF p>q {\) = s 9n(w)- — — . 

Note that 

V(a, b) = p + e(a, b) + //(a, b). 
By PP, the inverse image G\ of d = E/(ft) in K is isomorphic to the cover defined 

m ~ 

by the character det 2 . In particular, G\ is connected, hence the set E, occurring in 
Theorem 14.11 is a subset of the set of finite-dimensional irreducible representations 
of its Lie algebra u(n). 

Set f)i = f) n it(ra), f) 2 = b n «(p> <?)• 
Corollary 8.5 (Theta correspondence). W^i/i notation as in Theorem \4 . 1\ we have 

E = U ^n(-(pi)|^+£(a,b)). 

Moreover 

r(F n (e(a, b))) = ^ 2 (-(pi)| h2 + //(a, b)). 
Proof. Similar to the proof of Corollary 16.31 □ 

Let us finally display the character formula. We let W m _ d be the set of v £ W^_ d 

such that f (— (pi)|(, 2 +/i(a, b)-)-^^- 1 ) is regular for A(y4 p „! x For v £ W m _d, 

let 

Cu = C t,(-(pi)| te + M (a,b)+/i A «-i)- 
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From Proposition 18.41 we get that, if a G Pi max , b G Pj mjn then 
ch(L 2 (-( Pl ) lt)2 + /i(a,b))) = 

c v sgn{v)chV 2 {{v{- Pl + //(a, b) + p Am - x )} - p Am -'), 

while, if i G L with i < i max and a G Pj, b G P*, then 
ch{L 2 {-{ Pl )^ 2 + p(a,b))) = 

c„s#n(v)c/^ 2 0(-pi + p(a, b) + /A 1 " 1 )} - p^- 1 ). 

However we can check that if a G F* k , b G P^_j with k < i, then, v G W m _ d 

implies that W c i> = W c w with 10 G W 2 (i ' d ~ fe) , where W^ 4 '^ = {a G S* m | cr(e a ) = 
e a ,p — z + 1 < a < j> + d — fc}. It follows that the character formula can be 
written in a more uniform and symmetric way as follows: fix a G P£, b G P£ with 
k < p, h < q, and h + k < d. Choose a set W^Zd' d ~ k ^ of coset representatives for 
W c \W c W2 d ~ H,d ~ k) ■ Then 

c/i(L 2 (-(pi)|„ 2 +/i(a, b))) = 

c v sgn(v)chV 2 ({v(-( Pl ) lt)2 + /i(a, b) + p A —)} - p A ^). 

Once the character formula is written in this form, we can apply to it the argument 
given in Corollary 15.31 thus, with notation as in Section [5], we obtain 

Corollary 8.6. If k < p, h < q, k + h < d, a G F* k , b G ¥* h , and A = -(pi)|t, 2 + 
p(a, b) + p Am - x , then 

ch(L 2 (X ))= (-lY^ H V 2 ({w(X )}-p A —). 

A o 

9. On the Kac-Wakimoto conjecture 

As a final application of our denominator formulas we verify Kac-Wakimoto con- 
jecture in a remarkable special case. Let us recall briefly this conjecture. Let V 
be a finite dimensional irreducible highest weight module with highest weight A. 
Recall that the atypicality of A, denoted by atp(A) is the maximal number of lin- 
early independent mutually orthogonal isotropic roots which are orthogonal to A. 
The atypicality atpiV) is defined as the atypicality of A + p. The p-shift makes 
this definition independent of the chosen set of positive roots. Also recall that the 
supercharacter of V is defined as schV = Ylxeti* sdim(VA)e A . In [Hj the following 
conjecture is stated. 

Conjecture. There exists b G Q such that 

e P+A 

1L=i '(1-e A) 

where A is the highest weight of V and the set of simple roots contains mutually 
orthogonal isotropic roots (3\, . . . , (3 a tp(V) satisfying (p + A, /%) = 0. 
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We will refer to the latter condition as the KW condition. We prove this con- 
jecture when V is the natural representation of Lie superalgebras of type gl(m,n), 
D(m,n), D(n,m) with m > n and B(m,n), B(n,m) with m > n. Recall that, 
in these cases, we have atp(V) = min(m — l,n). Also note that atp(V) = m for 
B(m,m), and in this case KW condition does not hold for any choice of the set of 
simple roots, hence we exclude B(m,m) from our consideration. 

We fix the standard triangular decomposition of go and consider the sets of sim- 
ple roots compatible with this triangular decomposition. We embed our root sys- 
tem into the lattice spanned by {ej}™^ U {<5}™ =1 (with m > n); for example, for 
D(n,m),m > n, 2ej is an even root. We choose the inner product {ei,€j) = 5ij = 
-((>i,Sj)- 

9.1. Formula for a special root system. Consider a set of simple roots U which 
contains the set 

{^1 — ^2, ■ ■ ■ , €m-X — e m, Cm — — 82, ■ ■ ■ , ~ <5 n }. 

Then the corresponding total order on {e^}™ 1 U {5}" =1 is given by €\ > €2 > ■ ■ ■ > 
e m > 5i . . . > S n . For % — 1, . . . , n set 

7i := e m+l-i — 5{. 

Consider the natural representation V = V(e\). 
Set 

{li}i=i,..., n -i if m = n, 
{li}i=i,..., n if m > n. 
Note that T is a maximal set of isotropic pairwise orthogonal roots which are 
orthogonal to p + 1\. We have excluded B(m, m) since in this case the maximal set 
is {li}'i=^ U { e i + $m} and €\ + 5 m is "bad" in the sense that this root does not lie 
in a set of simple roots obtained from II by a series of odd reflections. Recall that 

We are going to show that there is a constant jy such that 

(9.2) Re"schV = ^(_^_ F ^). 

Write R = R /R u R t = UaeA+i 1 - 

9.1.1. As 0o _m °dule, V is the sum of two simple modules with highest weights Si 
and ei respectively. By the Weyl character formula one has 

(9.3) R e po schV = F w {e p0+tl - e p0+Sl ). 

9.1.2. Let q' be the simple subalgebra of q whose set of roots A' is the set of roots 
in A that are orthogonal to e\. We have that q' is of type glim — l,n),B(m — 
l,n),D(m — l,n),B(n, m — l),D(n,m — 1) respectively for g of type gl(m,n), 
B(m, n), D(m, n), B(n, m), D(n, m). Observe that, if S is a set of simple roots for 
g such that (ei, a) > for any aGE, then a set of simple roots for g' is given by 
£' = S\{ct G S I (a, e\) > 0}. In particular the set II' := II \ {ei — €2} is a set of 
simple roots for g'. We introduce p', p' , R' , R[, R', W for this root system in the 
standard way. Recall the denominator identity for g' in the form 
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with c = SZeQQI which can be rewritten as 

C s>' 

or 

^^) = ^( IXr6r ^ e - M) )- 

9.1.3. We now prove 

pPO+tl D 

(9-4) T w (e^ - ef*+*) = jV-M n (1 _ e - M) ) » 

where jy = - except for the case D(m,m) with ^ G A and in the latter case 
jv = Yc Note that, by (19. 3p . formula (19. 2 p is equivalent to formula (19.41) . 

Set " 

e po+ei i?i 

J * = M n«r(l-e-W) ) - 
Since -§f and e p0 ~ p ° are iV-invariant, one has 



/3eA+\A' 



so 



A=- c Yl (-l) |J| ^(e A ^), where A J = p + £i 

JcA+\A' /?eJ 

Now formula (19. 4p is equivalent to the formula 
(9.5) A = aJR w (e x * - e x ^~ s ^), 

where a = 1 except for the case D(m, m) with Si G A and a = 2 in the latter case. 

Recall that J-V(e A ) = if A is not regular. Let us find J such that Aj is regular. 
Write A = A £ [J Aj with A e in the span of {e i \™ =l and p = p e + p<5, where p e 
(resp., ps) is the standard "p" for A e (resp., for As). For any /3 G A+ \ A' one 
has (f3,ei) = 1, (f3,€i) — for i > 1. Thus Aj = p e + (1 — |J|)ei + fj, where z/j 
lies in the span of {<5j}™ =1 . The regularity of Aj is equivalent to the regularity of 
p e + (l — \ J\)ei and of Vj. 

For g/ m the element p e + (1 — |J|)ei is regular if and only if | J\ = 0, 1 or | J| > m, 
which is impossible since |A|" \ A'| = n < m. For \ J\ = 1 one has vj = ps + Si, 
which is regular only for % = 1. This gives (19. 5p . 

For B(m,n),m > n the root system A e (resp., As) is of type B m (resp., C n ). 
The element p e + (1 — | J|)ei is regular if and only if | J\ = 0, 1 or | J\ > 2m, which 
is impossible since |A^ \ A'| = 2n < 2m. For | J\ = 1 one has vj = ps ± Si, which 
is regular only for vj = ps + Si ; this gives (I9.5P . 
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For B(n,m),m > n the root system A e (resp., As) is of type C m (resp., B n ). 
The element p e + (1 — | J\)e\ is regular if and only if \ J\ = 0, 1 or \ J\ > 2m + 1, 
which is impossible because |A+ \ A'| = 2n + 1 < 2m + 1. Consider the case 
| J\ = 1. Then vj = ps if J = {ei} or vj = ps ± Si for J = {e± =F ^i}- Therefore 
Aj is regular for J = {ej, {ei - 5i},{e 1 + <5 n ,}. One has A {ei } = s Sn (X{ ei +s n }) so 
JV(e A{ei> + e A { £ i+^>) = 0. This establishes fl93|) . 

For D{n,m) the root system A e (resp., A^) is of type C m (resp., D n ). The 
element p e + (1 — | J|)ei is regular if and only if | J| = 0, 1 or | J| > 2m + 1, which is 
impossible since |A+ \ A'| = 2n < 2m + 1. For | J\ = 1 one has vj = ps ± Si and 
this element is regular only if vj = ps + Si] this gives (19.51) . 

For D(m,n),m > n the root system A e (resp., A^) is of type D m (resp., C n ). 
The element p e + (1 — | J|)ei is regular if and only if \ J\ = 0, 1 or | J| > 2m — 1, 
which is impossible since | A^ 1 " \ A'| = 2n < 2m — 1. For \J\ = 1 one has vj = p$±5i, 
which is regular only for vj = ps + <5i ; this gives (19.51) . 

Consider the case D(m,m) with A e (resp., A^) of type D m (resp., C n ). The 
element p e + (1 — | J\)ei is regular if and only if \J\ = 0, 1 or \ J\ > 2m — 1. As 
above, for | J\ — 1 the element uj is regular only if J — {e\ — 6i}, Vj — ps + S±. If 
\J\ = 2m, then J = A+\A' = {e 1 ±5 i }Z l so Aj = p + (1 -2m)e 1 = s ei s tm {p + e 1 ). 
If | J| = 2m - 1, then J = (A| \ A') \ {(3}, where j} = ex±8i and so v 3 = p 5 =F 5 4 
which is regular only if vj = ps + 5± and \ j = po + (2 — 2m) ei + 5i = s £l s £m (p + 
Since s ei s £m G we obtain 

A = 2F w (e p0+tl - e p0+Sl ). 

This establishes (I9.5P for this case. Having established (I9.5P in all cases, we have 
proven (19.21) . 

9.1.4. Let us deduce from (19. 2p Kac-Wakimoto formula (19.11) for the natural rep- 
resentation. If II is a set of simple roots for q and V^(A) is an irreducible finite di- 
mensional 0-module of highest weight A, recall that we say that the pair (II, V(A)) 
satisfies the KW condition if there are . . . , (3 a t P (v)} Q 5 with (3j) = for 
all i, j and (A + p^, = for all i. 

Assume that (II, V) satisfies the KW condition, where, as above, V is the natural 
representation of g. Since KW condition is obviously invariant under the action of 
the Weyl group we can consider the total order on {ej}™ 1 U {5«}" =1 corresponding 
to II. We can also assume that e\ > €2 > . . . > e m and 5\ > 62 > ■ ■ ■ > S n with 
£m-i$n > for g ^ gl(m,n) (this means that IT induces the standard triangular 
decomposition on $j )- 

Let A be the highest weight of the standard representation. Clearly, A is the 
maximal element in {e^}™^ U{<5j}™ =1 so A e {ei, 5i}. For gl(m, m),D(m, m) we may 
(and will) assume that e x > 5\ (since we can switch {e^} and {Si} in subsection 19. II) . 
Let us show that A = e\ for m > n. Indeed, for m > n one has atp(V) = n and so 
(A 7 #i) 7^ for some i. If A = Si, then (A + p, /%) = (Si, ^ which contradicts 
to the definition of Pi. Hence A = e\ > Si. 

Let a G II be a root satisfying (a, ei) = 1. Notice that (p, a) > so (p+€i, a) 7^ 
and thus a ^ {Pi}"=i V ^ ■ The set IT = n \ {a} is a set of simple roots for g' . Since 

def g' = atp{V), the set {eidi\ (3i = — di} is an arc diagram X for n'. Similarly, 
let X be the arc diagram for n' having the elements of T as arcs. Denominator 
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identity for g' gives, letting p = p n and p' = pg,, 

(9.6) atp(Vy. ■ ^( U _ r j'_ e - M) ) = ^(n-KV)J_ e -*,)- 

Let us show that p — p' = p — p'. Indeed, observe that p — p',p — p' are orthogonal 
to A'. Since e\ is maximal the inequality (a, e\) > for a G A forces a G A + n A + . 
Therefore 

aGA+\A' aeA+\A' aeA+\A' aeA+\A' 

= 2(p- p ,ei). 

Set £ = (p — p') — (p — p'). We conclude that £ is orthogonal to A' and e±. For 
7^ g/(m, n) this means that p — p' = p — p' . 

For gl(m,n) we obtain that £ is proportional to Y^T= 2 e * — SjLi The roots 
±(ei — 62) are the only roots in A \ A' which are not orthogonal to 62- Since 
Aq = Aq , one has 

2(p - p', e 2 ) = (ei - e 2 , e 2 ) = 2(p - p', e 2 ). 

Hence (£, €2) = so £ = as required. 
Substituting (19. 6 p in (I9.2p . we obtain, 

e p+ e i , e p' 

fle'sc/iV = jv?w{ ^ gr(1 _ e _ M) ) = J^/M/'(e p ' p+£1 ^( n _ e _ M) )) 

jv 



?wiw> [e^'^T w , (— _ -)) 



a^)i v w v nt p ra-e 

atp(V) ! w ^ nS (V) (! - e " ft ) ' 
This proves (19. ip with 6 = at ^yy ■ Using ( I1.13p . one can check that b has an 
expression depending only on atp(V). 
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